Correlation-Aware Exit Setting for Deep Neural Network Inference

Deep Neural Networks (DNNs) have been widely applied in video object recognition, yet tasks of varying complexity are often required to pass through the entire model, leading to inference redundancy for simpler tasks. By inserting exit branches at intermediate layers of the network, tasks that meet exit criteria can terminate early, reducing inference computation overhead and latency without significantly compromising accuracy. However, exit branches themselves introduce additional computational load, making it essential to strategically determine exit insertion points. The exit rules used in existing exit setting methods require tasks to sequentially attempt exits at each point, neglecting the correlation between adjacent video frames and resulting in numerous unnecessary exit attempts. To address this issue, we propose an exit rule based on the correlation between adjacent video frames, allowing video inference tasks to skip certain unsuitable exits. Building on this rule, we analyze the expected inference overhead under different exit settings and introduce an efficient estimation method. Finally, we present a high-efficiency exit setting method based on the ant colony optimization algorithm. Experimental results confirm that the proposed exit rule and setting method significantly reduce computational cost and inference latency.