ChannelZip: SLO-Aware Channel Compression for Task-Adaptive Model Serving on IoT Devices

Deploying deep neural networks (DNNs) on IoT devices for model serving is a promising solution for intelligent applications with high real-time requirements and bandwidth sensitivity. To cope with the prohibitive computation and storage overheads of modern DNNs, great efforts have been devoted to the model compression technique. Most existing model compression approaches focus on minimizing the model size and maximizing the average accuracy on all the inference tasks. However, real-world IoT tasks have various service-level objectives (SLOs). Models compressed by existing methods struggle to simultaneously meet SLOs in multiple dimensions, such as latency and accuracy. In this work, we study model compression with a joint consideration of SLO awareness and task adaptation. Through our extensive experience with model compression across various IoT tasks, we observe that the importance of individual channels in contributing to accuracy is heavily influenced by task-specific data distribution. Therefore, we design a channel Shapley algorithm to estimate the importance of individual channels in DNNs and propose a deep reinforcement learning based controller to incorporate SLOs into the compression objective. Integrating these designs, we propose and prototype ChannelZip, the first SLO-aware channel compression framework. Extensive evaluations on real IoT model serving systems show the effectiveness in task adaptation of ChannelZip. ChannelZip outperforms strong model compression baselines by 3.77% accuracy and achieves a 69% average parameter compression ratio. Real-world deployment on different IoT devices shows that ChannelZip meets all task SLOs and achieves up to 2.32 × inference speedup.