Real Time Network Threat Detection Using Machine Learning and Kafka in Middleware

The increasing complexity of network traffic resulting from internet-based services and data exchanges presents formidable obstacles to real-time analysis and decision-making in the field of network security. A real-time network traffic analysis system was created using Kafka, Redpanda, and machine learning to overcome these difficulties. Flow-based values were collected from CICFlowmeter. To efficiently classify network traffic, these attributes are subsequently examined using a machine learning model that has already been trained. Our integration of a Flask Application Programming Interface (API) with Kafka guarantees efficient, real-time data handling and processing. Efficient deployment is made possible via Docker, which also makes the system compatible with any existing server infrastructure. Performance analysis was conducted to assess how well the system handles real-time traffic and its accuracy in classifying network data, an accuracy of 99.4% is achieved using the Random Forest Classifier algorithm. The findings show that using Redpanda for Kafka, alongside machine learning, greatly improves the speed and effectiveness of data processing and classification. This approach provides a solid and adaptable solution for enhancing network security.