Enhancing Security and Reliability in Distributed Systems: A Hybrid Approach Integrating Snort Rules and Machine Learning for Anomaly Detection

Ensuring the security and reliability of distributed systems is a critical concern in today's digital landscape. In this research, we explore the integration of traditional Snort rules with advanced machine learning (ML) techniques for enhanced anomaly detection in network traffic. While Snort, a widely used Intrusion Detection System (IDS), excels in detecting known threats through predefined rules, it struggles with new and evolving attack patterns. To address this limitation, we propose a novel approach that represents Snort rules as features for ML models, specifically Isolation Forest and K-Means clustering, to identify anomalies. Our system captures and preprocesses network packets, dynamically extracting features based on Snort rules and transforming these features into a format suitable for ML analysis. Additionally, we introduce a feedback loop where anomalies detected by ML models are reviewed by security experts and used to refine the feature extraction process and retrain the ML models. This continuous learning process enhances the adaptability of the IDS, enabling it to respond to evolving threats in real-time. We evaluate the performance of the proposed models using the CICIDS2017 dataset, which provides a realistic distribution of normal and anomalous traffic. The results show that while Isolation Forest achieved an overall accuracy of 80% and K-Means achieved 79%, both models initially struggled with detecting anomalies due to imbalanced data. However, the introduction of dynamic feature extraction and the feedback loop mechanism significantly improved the detection rates, achieving a final accuracy of 84% for isolation forest and 86% for k-means. Isolation Forest is more optimal in terms of CPU and Memory resources, so we conclude that its well suited for packet processing systems which are typically resource-constraint. Through a detailed analysis, we also identify the strengths and weaknesses of each model, discuss the impact of advanced feature engineering and real-time implementation, and explore potential improvements through hybrid approaches.