Numerical Pruning for Efficient Autoregressive Models

Transformers have emerged as the leading architecture in deep learning, proving to be versatile and highly effective across diverse domains beyond language and image processing. However, their impressive performance often incurs high computational costs due to their substantial model size. This paper focuses on compressing decoder-only transformer-based autoregressive models through structural weight pruning to improve the model efficiency while preserving performance for both language and image generation tasks. Specifically, we propose a training-free pruning method that calculates a numerical score with Newton's method for the Attention and MLP modules, respectively. Besides, we further propose another compensation algorithm to recover the pruned model for better performance. To verify the effectiveness of our method, we provide both theoretical support and extensive experiments. Our experiments show that our method achieves state-of-the-art performance with reduced memory usage and faster generation speeds on GPUs.

Authors

Institutions

Publication Details

Journal
Proceedings of the AAAI Conference on Artificial Intelligence
Published
2025-04-11
DOI
https://doi.org/10.1609/aaai.v39i19.34249
Citations
4
Primary Topic
Matrix Theory and Algorithms
Type
article
Field-Weighted Citation Impact
8.72
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Numerical Pruning for Efficient Autoregressive Models

Jiuxiang Gu, Ryan A. Rossi, Bo Chen, Jing Liu et al.
4 citations
Proceedings of the AAAI Conference on Artificial Intelligence
Matrix Theory and Algorithms
8.72
article

Numerical Pruning for Efficient Autoregressive Models

Jiuxiang Gu, Ryan A. Rossi, Bo Chen, Jing Liu, Xipeng Shen, Yanzhi Wang, Xiang Chen, Tong Yu, Yufa Zhou, Ruiyi Zhang, T. Sun, Yufan Zhou, Zhao Song, Hao Tan, Pu Zhao
article en
4 citations

Abstract

Transformers have emerged as the leading architecture in deep learning, proving to be versatile and highly effective across diverse domains beyond language and image processing. However, their impressive performance often incurs high computational costs due to their substantial model size. This paper focuses on compressing decoder-only transformer-based autoregressive models through structural weight pruning to improve the model efficiency while preserving performance for both language and image generation tasks. Specifically, we propose a training-free pruning method that calculates a numerical score with Newton's method for the Attention and MLP modules, respectively. Besides, we further propose another compensation algorithm to recover the pruned model for better performance. To verify the effectiveness of our method, we provide both theoretical support and extensive experiments. Our experiments show that our method achieves state-of-the-art performance with reduced memory usage and faster generation speeds on GPUs.

Proceedings of the AAAI Conference on Artificial IntelligenceVol. 39(19)
Adobe Systems (United States) (US), Middle Tennessee State University (US), Australian Regenerative Medicine Institute (AU), California University of Pennsylvania (US), Monash University (AU), Universidad del Noreste (MX), University of California, Berkeley (US)
Openalex Percentile: Top 2%
Matrix Theory and Algorithms
8.72
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.