Sparse Sounds: Exploring Low-Dimensionality in Music Generation Model

We are the first to explore the intricacies of LLM compression techniques in the context of text-to-music generation, focusing on the MusicGen Transformer model. We implement and evaluate multiple compression methods, including pruning techniques and low-rank approximations to the MusicGen architecture. Additionally, we analyze the model’s sensitivity to calibration data and assess the impact of these pruning techniques on different music styles. Our results show that the SparseGPT pruning method is more robust than the Wanda method when using limited calibration music samples. Finally, we delve into transformer interpretability, investigating how specific layers within the MusicGen Transformer influence music style generation. We conclude that existing pruning methods can effectively reduce the MusicGen Transformer’s size, making it feasible for deployment in complex music generation models that require handling long sequences of notes.