LLM Compression for Enhanced Performance: A Comparative Study of Structured and Unstructured Pruning Techniques

The usage of transformer models has increased significantly in NLP tasks. However, deploying them in resource-constrained situations, including mobile devices and edge computing systems, is extremely difficult due to their high parameter count. Pruning is one of the prominent model compression strategies that offers an effective solution by reducing the model size and computational requirements without compromising its functionality. These methods are also perfect for real-time applications as they enable faster inference and consume less energy. This study investigates the comparative effectiveness of both structured and unstructured pruning methods towards compressing LLMs. Using the GLUE benchmark, performance is evaluated across tasks using accuracy, F1 score, Matthews correlation coefficient (MCC), and model size. The results demonstrate significant differences between structured and unstructured pruning methods, which provides valuable insights for optimizing transformer models in real-life applications. Structured pruning, in particular, shows stronger resilience to parameter reduction and consistently delivers better results at higher pruning ratios, making it advantageous for latency-sensitive, real-time applications. Furthermore, a pruning ratio of 0.4 was found to preserve most of the model’s original accuracy, while 0.6 offers a balanced compromise between compression efficiency and performance retention.