Sparse Ternary Matrix Multiplication with Tensor Core for Transformer

The Transformer architecture, despite its scaling law, faces expensive computational cost challenges as the number of parameters increases. Quantization methods like Ternary-BERT and BitNet address this issue using ternary matrices for weight parameters. While most research focuses on accelerating ternary matrix multiplication (TMM) on specific hardware such as FPGAs, our work aims to accelerate TMM on GPUs by co-designing around the characteristics of sparse ternary matrices and GPU architecture. In this paper, we propose two TMM methods that leverage the performance of CUDA Cores and Tensor Cores, respectively. We demonstrate that the proposed methods outperform dense matrix multiplication at sparsity levels of about 88% and above.