Astrolabe: Optimising Edge Inference through Multi-Depth Model Orchestration

The deployment of image classification algorithms, particularly for processing video streams, is increasingly shifting to edge devices. This shift to edge devices enhances privacy, conserves bandwidth, and reduces latencies. In edge environments, dynamic responsiveness is crucial to mitigate task collocation, load peaks, and hardware/network outages. Orchestrators address these challenges via scaling and scheduling. This work proposes integrating domain-specific properties for orchestration, introducing machine learning model depth as a novel option. Using the Ecoscape Benchmark, we benchmark our proposed orchestrator, Astrolabe, against the Kubernetes Horizontal Pod Autoscaler. We conduct experiments exploring the impact of faults and changing workloads on predefined Service Level Objectives. The results show that the orchestration action of adjusting the model depth is promising in maintaining SLOs.