Lightweighting Large Models: A Review of Model Compression Techniques

With the rapid development of deep learning technology, neural network models have achieved remarkable results, but their large number of references and high computational resource requirements limit the wide range of applications.Therefore, model compression techniques emerge, aiming to reduce computational complexity, memory occupation, and energy consumption overhead of the models to meet the practical deployment requirements without sacrificing performance. In this paper, we systematically sort out the research results in the field of model compression in recent years. Firstly, we introduce the definition and goals of model compression, and discuss its challenge and evolution of the techniques. Then a variety of mainstream compression methods are elaborated in detail, including pruning, quantization, knowledge distillation, lowrank factorization, and parameter sharing, and analyze their principles, strengths, weaknesses, and applicability scenarios. We also provide a comparative analysis of these methods, highlighting their trade-offs in terms of compression rate, performance, and computational efficiency. In addition, we introduce commonly used datasets, evaluation metrics, and future research directions for model compression in image classification and natural language processing tasks. By comprehensively summarizing and analyzing the existing work, we expect to provide model compression researchers with a clear technology mapping and research roadmap to promote their researches.