Empowering Resource-efficient MoE Co-Adaptation for Multi-Task Vision Systems

Mobile and IoT vision applications increasingly utilize multitask deep learning (DL) models for real-time inference. The integration of Mixture of Experts (MoE) and Vision Transformers (ViTs) is particularly effective due to the task-agnostic backbone and scalable task-specific heads. However, data drift in open-world environments can lead to accuracy drops and safety risks, as observed in systems like Google Waymo One and NVIDIA NoTraffic. We present AdaSprite to integrate ViT-based MoE co-adaptation into resource-limited IoT systems, first addressing the expert dynamic sparsity during retraining as an opportunity in multi-task settings through cross-task collaboration in computation, I/O, and resource scheduling. Implemented as microservices, AdaSprite improves adaptation accuracy by 34.5% and latency by 77.1%, outperforming baselines across four practical scenarios.