GASA: Rank-Sliced GAther-Scatter Activations and Application to Sparsity-Preserving Parameter-Efficient Fine-Tuning

We present a novel rank-sliced GAther-Scatter Activation (GASA) algorithm to minimize I/O costs in computing neural network layer activations (XW) between a data matrix X, and a singular value decomposition (SVD) of a weight matrix W = UΣV<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">T</sup>. We maintain high accuracy with ResNet-18 [1] on the CIFAR-10 [2] dataset and Deberta-V3-base [3] on the IMDB dataset [4] at high sparsities (i.e., up to 85% sparsity for U and V ) by using rank pruning for W and UV pruning for each rank slice. Furthermore, with rank-sliced computation, we can perform parameter-efficient fine-tuning on the resulting sparse networks while preserving sparsity to retain the sparsity-induced computational efficiency for inference. That is, our rank-sliced weight update preserves the original sparsity structure of each W. Our sparsity-preserving fine-tuning maintains model accuracy under adapter ranks as low as 8, compared to the rank of 150 of the pre-trained pruned model.