A Thread-level Stream Scheduling Method for Accelerating LVMs' Inference on a Resource-constrained Platform

As a new generation of edge devices, the integrated CPU/GPU architecture has opened up new opportunities for deploying different scale vision models. In order to reduce models’ inference time on the integrated devices, this paper first compresses deep learning models using model quantization. The quantization process greatly reduces the computation requirements of a model, which enables its deployment on embedded development boards. However, quantization also leads to lower GPU resource utilization during inference on integrated devices. This insufficient utilization results in slower inference speed. To address this problem, this paper first depicts the data flow of model inference within an integrated device. Secondly, this paper implements a unified memory management between the CPU and GPU based on managed memory strategy. Finally, this paper designs a thread-level stream scheduling method to improve GPU utilization and throughput during model inference in a pipeline way. Experimental results show that the proposed method achieves a 2x-10x improvement in throughput compared to the TensorRT’s default scheduling method, which is crucial for realizing real-time inference tasks on edge devices.