Advancing Network Security: Attack Response, Anomaly Detection, and Domain-Adaptive Intrusion Detection Systems

Digital network expansion necessitates enhanced security mechanisms against cyber threats, particularly Distributed Denial-of-Service (DDoS) attacks and anomalies across network environments. This dissertation presents systematic advancements in network intrusion detection systems (NIDS) through three interconnected research phases: attack source identification, anomaly detection, and domain adaptation. The first phase of this dissertation introduces a novel Autonomous System (AS) traceback mechanism, addressing IP spoofing challenges in network attacks by efficiently identifying attack origins with minimal router involvement. By encoding AS numbers into IP packet headers using probabilistic packet marking, the method enables efficient reconstruction of attack paths. The second phase develops advanced anomaly detection models utilizing contractive autoencoders as the core architecture. This begins with a model specifically designed for DDoS detection, employing contractive autoencoders to capture normal network traffic patterns. An iterative threshold optimization enhances the model’s precision in distinguishing between benign and malicious activity. The next model targets the distinct challenges in IoT security, such as device heterogeneity and the lack of a unified anomaly detection objective. By integrating Deep SVDD with a Contractive Autoencoder, this approach enhances latent feature representations to improve the relevance of these representations for anomaly detection, enabling the identification of both known and unknown threats within IoT environments. The final model in this phase combines Contractive Autoencoders with K-means clustering, applying optimized hyperparameters to enhance anomaly detection across diverse network scenarios. The third phase of this dissertation focuses on domain adaptation to maintain detection effectiveness across diverse network environments. This model incorporates a Convolutional Neural Network (CNN) as a feature extractor, alongside a Gradient Reversal Layer (GRL), to learn domain-invariant features. Trained on labeled source data, with a small subset of labeled target data and a larger pool of unlabeled target data, this approach demonstrates consistent performance across different dataset pairs, addressing the challenge of domain shifts in NIDS and ensuring robustness across varied network conditions.