MTDecipher: robust encrypted malicious traffic detection via multi-task graph neural networks

Abstract The widespread adoption of encrypted traffic protocols has significantly increased the challenge of detecting malicious traffic. Existing detection methods based on deep learning typically rely on fine-grained features of data packets, such as length sequences and intra-flow interaction graphs. However, these features are highly susceptible to disruption by diverse network environments and traffic obfuscation. This paper proposes MTDecipher, a robust method for detecting encrypted malicious traffic based on multi-task Graph Neural Network (GNN). MTDecipher employs a bidirectional attentive sequence encoder to mitigate the impact of diverse network environments and traffic obfuscation on packet length sequences, along with an edge-block dual sampling method and a multi-task GNN model to mitigate the training bias introduced by the unbalanced distribution of traffic. In the bidirectional attentive sequence encoder, a combination of a Bi-GRU layer and an attention pooling layer is utilized to enhance the bidirectional encoding by generating weights for each element in the sequence, thereby obtaining robust encrypted traffic sequence features. In the edge-block dual sampling method, two rounds of sampling are involved to generate more evenly distributed subgraphs as training data, which reduces the local structural bias resulting from the aggregation of malicious flows. In the multi-task GNN model, the losses for both edge and node classification tasks are simultaneously optimized, thereby minimizing the homogeneity of adjacent edges. Experimental results on two real-world datasets with traffic obfuscation demonstrate that MTDecipher outperforms eight existing methods in terms of effectiveness in detecting encrypted malicious traffic.