Open Set Recognition of Malicious Traffic Based on Spatial Orientational Location Constraint

Malicious traffic identification is crucial for mitigating malware threats. However, most existing approaches rely on closed-set recognition (CSR), where classifiers can only categorize samples within a predefined set of known categories. Additionally, many studies focus on extracting statistical features from network traffic while overlooking valuable information in raw data files. In real-world scenarios, malicious traffic identification should be treated as an open-set recognition (OSR) problem, where models must detect both known and previously unseen malware categories. We propose a spatial orientational location constraint-based method for the open-set identification of malicious traffic (MOSCPL). This method splits open-set malicious traffic detection into two parts: classification of known classes and identification of unknown classes. We reduce open space risk by designing spatial orientational location constraint loss (SOLCL)while improving the accuracy of known class classification and unknown class identification. To leverage raw data more effectively, we transform preprocessed PCAP files into images, allowing automatic feature extraction from original network traffic. Finally, we validate our method by training on the MCFP dataset and testing on multiple non-homologous datasets, demonstrating its effectiveness in detecting malicious traffic.