Machine Learning for Intrusion Detection in Network Traffic

Network intrusion detection is a critical safeguard for modern infrastructures. Traditional rule-based and static signature-based systems rely on predefined attack patterns and are effective against known threats. However, they struggle with novel attacks and can generate excessive false alarms. Machine learning (ML) has been widely proposed and applied to enhance intrusion detection by enabling systems to learn patterns in network traffic and identify anomalies beyond predefined signatures. Building on this foundation, this thesis evaluates which category (supervised or unsupervised) and which models within each category perform best for intrusion detection. It evaluates four models, including two supervised classifiers, Support Vector Machine and Random Forest, and two unsupervised detectors, an autoencoder and the HDBSCAN algorithm. Each model is trained and tuned on the CIC IDS 2018 benchmark dataset and then tested on unseen traffic, including simulated zero-day events. Key metrics such as accuracy, precision, recall, F1 score, and false alarm rate are recorded. Results are compared based on overall performance, with one attack type excluded from training to assess resilience against entirely new threats. The analysis provides a neutral ranking of the evaluated models and shows that the most effective strategy is a hybrid, two-layer IDS: a high-precision Random Forest (or similar ensemble) handles the bulk of traffic, while a lightweight autoencoder runs in parallel to flag outliers that deviate from known patterns. This arrangement preserves the near-perfect precision of supervised detection, adds a safety net for emerging threats, fits comfortably on commodity hardware with room for future scale-out, and therefore represents the most practical path forward for a machine-learning-based intrusion-detection system.