Performance Comparison of Tensor Decomposition on LLMs

This study explored using tensor decomposition techniques, specifically canonical polyadic decomposition (CPD) and Tucker decomposition, for optimizing transformer-based large language models (LLMs).By focusing on feed-forward layers and the final fully connected layer, we evaluated the impact of these decompositions on memory efficiency, computational performance, and overall model behavior.Experiments on the NSMC and Tiny Shakespeare datasets were used to analyze the trade-offs between compression, training time, and accuracy for different rank settings.The results reveal that Tucker decomposition provides a balanced approach to compression and computational efficiency, while CPD offers competitive performance but with higher computational costs.This work provides valuable insights into deploying tensor decomposition techniques for optimizing LLMs in resource-constrained environments.