Robust Detection of Malicious Encrypted Traffic via Contrastive Learning

Traffic encryption is widely used to protect communication privacy but is increasingly exploited by attackers to conceal malicious activities. Existing malicious encrypted traffic detection methods rely on large amounts of labeled samples for training, limiting their ability to quickly respond to new attacks. These methods also are vulnerable to traffic obfuscation strategies, such as injecting dummy packets. In this paper, we propose <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">SmartDetector</i>, a robust malicious encrypted traffic detection method via contrastive learning. We first propose a novel traffic representation named Semantic Attribute Matrix (SAM), which can effectively distinguish between malicious and benign traffic. We also design a data augmentation method to generate diverse traffic samples, which makes the detection model more robust against different traffic obfuscation strategies. We propose a malicious encrypted traffic classifier that first pre-trains a model via contrastive learning to learn deep representations from unlabeled data, then fine-tunes the model with a supervised classifier to achieve accurate detection even with only a few labeled samples. We conduct extensive experiments with five public datasets to evaluate the performance of SmartDetector. The results demonstrate that it outperforms the state-of-the-art (SOTA) methods in three typical scenarios. Specifically, in the evasion attack detection scenario, SmartDetector achieves an F1 score and AUC above 93%, with average improvements of 19.84% and 18.17% over the SOTA method, respectively.