Hybrid Coarse-Grained Transprecision boosted by a Custom GEMM Kernel to accelerate Parametric Linear Systems

Abstract Processing parametric linear systems of the form \((G(P):=X^\top P X)\) is computationally expensive due to the large size of matrix \((X)\) and reliance on double-precision floating-point arithmetic. As many applications can tolerate small numerical errors, performance trade-offs can achieve low-precision format GPU solutions. A hybrid coarse-grained transprecision computing framework is proposed that comprises an offline phase that identifies suitable low-precision formats for matrices \((X)\) and \((P)\) under a predefined error constraint, and an online phase that dynamically analyzes each incoming \((P)\) to select the optimal precision for computing \((G(P))\). It allows a \((21.6M \times\ 500)\) matrix, \((X)\), to be processed up to 58x faster than its double-precision version, enabling real-time evaluation of 2 \((P)\) matrices/s on an NVIDIA A100 GPU, with an average relative error below 0.6%. In addition, a fused custom GEMM kernel for computing \((G(P):=X^\top P X)\) has also been developed, by integrating two standard CUDA-based BLAS operations into a single CUTLASS-based kernel. It eliminates intermediate memory transfers and reduces computational overhead, providing an additional 3x speedup over the conventional implementation.