A Dynamic Distributed Scheduler for DNN Inference on the Edge

Edge computing plays a pivotal role in IoT applications that require rapid and secure data processing. How-ever, these applications are typically resource-demanding, and the resources available at the edge are often significantly less than those available in cloud environments. Thus, optimally utilizing these constrained resources is key to fulfilling various application requirements such as low latency, privacy, and cost-effectiveness. Given the dynamic and hybrid characteristics of IoT environments, static scheduling approaches frequently fail to adequately address these complex and varying demands. This paper presents the design, implementation, and evaluation of a dynamic distributed scheduler tailored for DNN inferences on edge computing platforms, based on sophisticated sub-model profiling techniques. This scheduler's core feature is its ability to continuously monitor the state of the IoT infrastructure, dynam-ically adapting the distribution of computing tasks according to real-time environmental and network conditions. Diverging from previous research that typically segments DNN models into two parts, this study proposes a more granular approach that splits models into multiple segments, thereby maximizing the utilization of diverse edge computing resources. The implementation of this scheduler, including integration with a GPU, demonstrates its effectiveness across various DNN models, highlighting its practical utility in leveraging edge computing capabilities.