Exploiting Operator-Level Concurrency Control to Guide Deployment for Real-Time Tasks in Edge AI Cluster

Existing task deployment frameworks for edge clusters optimize at the model-level, lacking fine-grained resource awareness and concurrency control, where the urgent tasks are frequently blocked and miss their deadlines. Therefore, we propose a multi-level collaborative deployment framework Coconut for real-time deep learning tasks in the typical heterogeneous edge GPU cluster. Coconut collaboratively optimizes model deployment and fine-grained concurrency control. To address the high complexity of multi-level collaborative optimization, we employ an efficient learning-based search algorithm. Based on the operator-level information, we also pre-train an accurate latency predictor for each device, enabling centralized optimization to further accelerate the search. We conduct a preliminary evaluation in an edge cluster and validate the effectiveness of Coconut.