Early Prediction of Dynamic Sparsity in Large Language Models

Large language models are powerful but computationally very expensive.We investigate dynamic sparsity in attention mechanisms, using the OPT model as a case study.We explore the dynamic nature of redundancy in attention heads and analyze which components of the model provide sufficient information to predict sparsity effectively.Our findings highlight the norm of attention outputs as a reliable criterion for ranking head importance.We systematically evaluate embeddings across layers and time steps, showing that dynamic sparsity predictions can be achieved early in the model pipeline with minimal loss in accuracy.By elucidating the mechanisms underlying dynamic sparsity, this work lays a foundation for more efficient and scalable transformer models.