Informed Pruning of Embedding Dimensions in Large Language Models

This paper investigates the effectiveness of pruning Large Language Models (LLMs) along embedding dimensions to improve efficiency while maintaining performance for specific downstream tasks. It describes methods for selecting and pruning less important embedding dimensions and associated parameters based upon either variance or principal component analysis.Ablation experiments are performed to evaluate the pruning method. Results show comparableeffectiveness to earlier structural pruning methods. However, it is determined that variance byitself is not a good metric to measure the importance of embedding dimensions.