Demo: CollabTrans: Device-cloud Collaborative Inference Framework for Transformer-based Models

Model splitting and offloading part of the DNN model from the mobile device to the cloud server, known as collaborative inference, improves end-to-end latency and server throughput in CNN-based models. However, current collaborative approaches do not apply to popular Transformer-based models, as these models generate large intermediate outputs with significant transmission latency and have a uniform block structure that makes it challenging to serve tail models efficiently on the server. We propose CollabTrans, a collaborative inference framework designed for Transformer-based models to enhance server scalability when handling requests from numerous end devices, taming the large output with truncated SVD and serving heterogeneous tail models by sharing a complete model. We demonstrate our framework with a real-time image classification mobile application together with background inference request traffic, showing the high inference throughput of our framework.