Next Generation Linear Algebra: One Singular Value Decomposition to Catch Them All across Data Size, Precision, and Hardware

The Singular Value Decomposition (SVD) is a foundational building block in many applications, including low-rank adaptation (LoRA) for large language models (LLMs). Historically, separate SVD implementations have been designed for each data precision, for each hardware vendor, and for each hardware type (personal computer and HPC). This divergence leads to increased development time, the need to redevelop entire libraries when new architectures or data types emerge, and significant complexity for the end user. In this abstract, we discuss a work in progress to develop an alternative: a unified SVD, enabled by abstraction layers. We demonstrate that state-of-the-art performance across the board can be reached using abstraction frameworks, and investigate the performance engineering process and the characteristics that enable adaptable performance.