Efficient Compression of Large Language Model based on Hybrid Layer and Channel Pruning

Large Language Models (LLMs) are constrained by their large parameter size and high computational resource consumption, making deployment on terminal devices with limited computing resources difficult. This severely hinders the deployment and application of LLMs in power grid systems. To address this, we propose a hybrid structured pruning method based on layer and channel pruning to compress LLMs, enabling their efficient deployment in power grid systems. Our method first removes noncritical layers based on inter-layer cosine similarity, thereby achieving a significant reduction in model parameters. Then channel-pruning based on PCA is employed, aiming to improve model inference speed while preserving performance. Experimental results on Wikitext2 and PTB demonstrate that at a 30% pruning rate, our method exhibits a superior performance retention compared to single-layer pruning and channel-wise pruning methods. For generation tasks, the model inference speed is increased by 5.91%. Our work provides a novel insight for future research on structured pruning, and inspires more research on composite pruning from multi-dimensional perspectives.