ELPG: End-to-End Latency Prediction for Deep-Learning Model Inference on Edge Devices

The demand for running deep learning models on edge devices is increasing. In order to achieve lower latency and higher accuracy on edge devices, many studies have custom-designed deep learning models for edge devices, among which hardware-aware neural architecture search (HW - NAS) is the most successful. HW-NAS require measuring the model's in-ference latency on real edge devices to guide the design of model structures. Directly measuring inference latency on edge devices is prohibitively expensive, leading many works to predict model inference latency using various methods. In this paper, we propose ELPG, an end-to-end inference latency predictor based on Graph Neural Networks and Gate Recurrent Unit. ELPG represents a model as a kernel sequence, and use this sequence as input to predict the model's inference latency. ELPG has no restrictions on the structure of the model and can predict the inference latency of the current advanced CNNs and their variants. Experimental results indicate that ELPG achieves higher prediction accuracy compared to existing methods, and with only 100 latency data, ELPG achieves 82.15% accuracy on Jetson Nano.