Opara: Exploiting Operator Parallelism for Expediting DNN Inference on GPUs

GPUs have become the <i>defacto</i> hardware devices for accelerating Deep Neural Network (DNN) inference workloads. However, the conventional <i>sequential execution mode of DNN operators</i> in mainstream deep learning frameworks cannot fully utilize GPU resources, even with the operator fusion enabled, due to the increasing complexity of model structures and a greater diversity of operators. Moreover, the <i>inadequate operator launch order</i> in parallelized execution scenarios can lead to GPU resource wastage and unexpected performance interference among operators. In this paper, we propose <i>Opara</i>, a resource- and interference-aware DNN <u>Op</u>erator <u>para</u>llel scheduling framework to accelerate DNN inference on GPUs. Specifically, <i>Opara</i> first employs <monospace>CUDA Streams</monospace> and <monospace>CUDA Graph</monospace> to <i>parallelize</i> the execution of multiple operators automatically. To further expedite DNN inference, <i>Opara</i> leverages the resource demands of operators to judiciously adjust the operator launch order on GPUs, overlapping the execution of compute-intensive and memory-intensive operators. We implement and open source a prototype of <i>Opara</i> based on PyTorch in a <i>non-intrusive</i> manner. Extensive prototype experiments with representative DNN and Transformer-based models demonstrate that <i>Opara</i> outperforms the default sequential <monospace>CUDA Graph</monospace> in PyTorch and the state-of-the-art operator parallelism systems by up to <inline-formula><tex-math notation="LaTeX">$1.68\boldsymbol{\times}$</tex-math></inline-formula> and <inline-formula><tex-math notation="LaTeX">$1.29\boldsymbol{\times}$</tex-math></inline-formula>, respectively, yet with acceptable runtime overhead.