Real-Time Visualization and Detection of Malicious Network Flows in IoT Devices using a Scalable Stream Processing Pipeline

The continuous adoption of Internet of Things (IoT) devices has established notable security risks. In recent times, IoT devices are often targeted by malicious actors as IoT devices are limited with processing and memory resources. The detection of malicious network flows in real-time is necessary to maintain the security of IoT networks. This paper addresses the challenge by presenting a scalable stream processing pipeline that identifies malicious and non-malicious flows from IoT devices in real-time. The proposed system utilizes NFStream to generate network flows from device packets. The generated network flows are then published to a Kafka topic and consumed by a Kafka consumer. A distributed random forest-based machine learning model from Apache Spark MLlib is trained to classify these flows. The classification results are stored alongside the flows in a MongoDB database. Real-time visualization of the count of malicious and non-malicious flows for each IoT device is provided via Streamlit. Experimental results demonstrate the system's capability to accurately classify network traffic in real time while maintaining scalability for large-scale IoT environments. The proposed pipeline offers a robust solution for real-time malicious flow detection in large-scale IoT networks.