Accelerating dilated Winograd convolution with fused GPU kernel using tensor cores

Abstract 2D convolutional neural networks (CNNs) achieve remarkable accuracy across diverse computer vision tasks, yet their inference efficiency remains suboptimal. As a fast convolution algorithm, Winograd convolution can significantly accelerate convolution operations, which constitute the primary performance bottleneck in CNNs. However, existing GPU-based implementations are restricted to accelerating standard convolution with unit dilation, while lacking support for dilated convolution. We propose TC-DWC, a GPU-based 2D Winograd convolution implementation tailored for dilated convolution, which effectively accelerates dilated convolution tasks across various configurations. TC-DWC employs a two-step tile reorganization scheme that reconciles the incompatibility between Winograd convolution and dilated convolution. Moreover, it leverages specialized high-throughput tensor cores (TCs), replacing conventional vector units to accelerate the computationally dominant matrix multiplications. To alleviate TC-DWC’s substantial memory footprint, we further develop a multi-stage kernel fusion strategy that eliminates intermediate array allocations, yielding Fused TC-DWC. Experimental results demonstrate that, for single convolutional layers, TC-DWC and Fused TC-DWC deliver average speedups of $$1.56\times $$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mn>1.56</mml:mn> <mml:mo>×</mml:mo> </mml:mrow> </mml:math> (up to $$2.45\times $$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mn>2.45</mml:mn> <mml:mo>×</mml:mo> </mml:mrow> </mml:math> ) and $$1.29\times $$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mn>1.29</mml:mn> <mml:mo>×</mml:mo> </mml:mrow> </mml:math> (up to $$1.80\times $$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mn>1.80</mml:mn> <mml:mo>×</mml:mo> </mml:mrow> </mml:math> ) over cuDNN optimal implementation, respectively, with Fused TC-DWC reducing memory footprint to 35% of TC-DWC and 55% of cuDNN. For end-to-end inference of CSRNet and DLinkNet34, TC-DWC and Fused TC-DWC deliver average speedups of $$1.38\times $$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mn>1.38</mml:mn> <mml:mo>×</mml:mo> </mml:mrow> </mml:math> (up to $$1.41\times $$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mn>1.41</mml:mn> <mml:mo>×</mml:mo> </mml:mrow> </mml:math> ) and $$1.20\times $$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mn>1.20</mml:mn> <mml:mo>×</mml:mo> </mml:mrow> </mml:math> (up to $$1.22\times $$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mn>1.22</mml:mn> <mml:mo>×</mml:mo> </mml:mrow> </mml:math> ) over PyTorch baseline, respectively, with Fused TC-DWC enabling support for larger batch sizes.