TINIEE: Traffic-Aware Adaptive In-Network Intelligence via Early-Exit Strategy

In-network (or on-path) inference over programmable data planes (PDPs) allows fast and low-overhead inference using deep neural networks (DNN). To alleviate the massive processing and deployment cost of in-network inference, distributed deployment on multiple programmable network devices is mainly adopted. However, it is likely to produce a considerable amount of network traffic due to the exclusive forwarding chain and intermediate data between submodels. In this work, we propose a traffic-aware adaptive in-network inference scheme, TINIEE, to maximally reduce the network traffic of in-network inference without causing a significant reduction in classification performance. To this end, we first devise an adaptive inference method on the data plane striking the balance between the classification performance and the network traffic cost. Furthermore, we formulate a traffic minimization problem to decide the proper location of each submodel considering each flow's exit tendency with a predefined confidence threshold. Since the problem is excessively complicated, we devise a low-complexity practical submodel placement algorithm. We implement the proposed scheme on software-programmable switches, and the evaluation results demonstrate that TINIEE reduces network traffic by up to 34.48 % compared to the state-of-the-art, while maintaining sufficiently high classification performance.