Scalable Streaming Data at the Egde Using Deep Learning

ABSTRACT In recent years, the proliferation of IoT devices and edge sensors has led to an exponential increase in real-time streaming data, challenging conventional deep learning methods designed for static, centralized datasets. ScaDLES, addresses these challenges by introducing a framework for training deep learning models directly on edge devices. The primary objective is to overcome issues such as heterogeneous streaming rates, limited memory and storage, and high communication overhead typical in distributed environments. The system is implemented using Python with PyTorch for model training, Apache Kafka for simulating streaming data, and Docker containers to emulate a distributed edge environment. it indicates that ScaDLES significantly enhances training speed and reduces resource consumption compared to traditional distributed deep learning frameworks. It’s not only improves the feasibility of real-time model updates on the edge but also paves the way for applications in surveillance, autonomous vehicles, and smart cities. Keywords: Edge Computing, Streaming Data, Deep Learning, Distributed Training Gradient- Compression.