TransGraphNet: robust detection of malicious encrypted network traffic <i>via</i> transformer and graph neural models

As encrypted network traffic becomes more prevalent, cybercriminals increasingly conceal their malicious activities within encrypted session contents. Traditional methods for detecting malicious encrypted traffic focus on inspecting the plaintext payload during the Transport Layer Security (TLS) handshake phase and analyzing directed Internet Protocol (IP) packet length sequences during the subsequent encrypted transmission phase. However, these methods often have high miss-detection rates in real-world scenarios. For example, attackers employ active traffic obfuscation techniques, such as forging TLS Service Name Indicators (SNI) and certificates to evade detection, while network environments introduce passive obfuscation through packet sequence perturbations. To address these challenges and improve detection robustness, we propose transGraphNet, a novel framework that integrates multi-granularity features from both the packet-length sequences within individual network connections and the flow-relation graphs representing interactions among multiple connections. For each session between a pair of client-server IP hosts, we partition its involved flows into session windows using an adaptive session window algorithm, and then capture concurrency and trigger relationships between their transmitted flows by constructing an Adaptive Sliding-Window Flow-level burst Graph (AFG). Unlike prior methods such as FG-Net, which constructs a flow-relation graph for each client host, our TransGraphNet groups the flows within the same client-server session and with proximate start times into a session window, which is used as a more fine-grained target for traffic classification. Additionally, for each five-tuple flow, we apply the packet length standardization using an enhanced Power Law Division (PLD) algorithm to mitigate time-series noise caused by passive obfuscation. We also integrate packet length sequence modeling and capture its long-range dependencies through a transformer-GNN model, thereby enhancing its representational power. Specifically, for each flow, temporal statistical features are extracted using Convolutional Neural Network (CNN), while long-range dependencies within the packet length sequence are captured via Bidirectional Encoder Representations from Transformers (BERT). These flow representations serve as node features in the AFG, where a graph attention network is employed to propagate and aggregate information across the graph’s topology. The resulting AFG representation is then used for malicious encrypted traffic detection. Extensive experiments on the traffic dataset with obfuscation show that TransGraphNet outperforms state-of-the-art methods such as FG-Net, ET-BERT, and GraphDApp, improving accuracy by 11%, 5.4% and 6.8%, respectively.