A Generalized and Real-Time Network Intrusion Detection System Through Incremental Feature Encoding and Similarity Embedding Learning

Many <i>Network Intrusion Detection Systems</i> (NIDSs) process sessions only after their completion, relying on statistical features generated by tools such as CICFlowMeter. Thus, they cannot be used for real-time intrusion detection. Packet-based NIDSs address this challenge by extracting features from the input packet data. However, they often process packets independently, resulting in low detection accuracy. Recent advancements have captured temporal relations between the packets of a given session; however, they use a fixed window size for representing sessions. This representation is inefficient and ineffective for processing short and long sessions. Moreover, these systems cannot detect unobserved attack types during training. To address these issues, the proposed method extracts features from consecutive packets of an ongoing session in an online manner and learns a compact and discriminative embedding space using the proposed multi-proxy similarity loss function. Using the learned embedding and a novel class-wise thresholding approach, our method alleviates the imbalance issue in NIDSs and accurately identifies observed and novel attacks. The experiments on two large-scale datasets confirm that our method effectively detects attack activities by processing fewer than seven packets of an ongoing session. Moreover, it outperforms all the competing methods by a large margin for detecting observed and novel attacks.