Harnessing self-supervised learning to boost malicious traffic detection with enhanced attention

The existing deep learning-based malicious traffic detection methods generally suffered from three main problems: labeled sample scarcity, inadequate representation of malicious behavior traffic features, and a high false positive rate due to ineffective integration of behavioral association patterns during detection. To address these issues, an end-to-end malicious traffic detection method named MTAttention was proposed. The heterogeneous header features and payloads of network behavior traffic were unified and encoded, resulting in a standardized flow and session representation at the packet level through structured multi-packet sequential traffic representation. Based on the MAE model, a self-supervised mask pre-training strategy was adopted, and a visual Transformer was utilized to extract rich traffic feature representations. By selectively focusing on the spatial and intervariable dependencies among different parts of the input packet sequence, a general traffic representation was learned, and the weight parameters of the encoder model were used to initialize subsequent tasks, which accelerated the model training process. A packet sequence feature fusion strategy based on channel attention was introduced, and a multi-dimensional attention mechanism, along with labeled data, were used to fine-tune the model’s weights to adapt to traffic identification and classification tasks. Before making classification decisions, the integration ability of high-weight features was enhanced to further improve the model's detection precision. The CIC-IDS2017 dataset was employed for the experiment, and the results showed that in the multi-classification scenario for malicious traffic identification, the precision of the MTAttention method could reach an average of 98.7%, and the inference efficiency exceeded 1590 samples per second. Compared with Flow-MAE, an improvement over the MAE paradigm, MTAttention maintained high precision while requiring only 1.56% of the parameter count and 63.89% of the memory overhead of Flow-MAE. Additionally, the average inference speed enhancement was approximately 100%, and the model size was only 5.17 MB.