Sustainable LLMs: Compression-Driven Efficiency for Faster Training and Inference

Large Language Models (LLMs) with billions of parameters present substantial challenges in terms of memory, computation, and energy requirements. Their scale often renders them impractical for deployment on resource-constrained devices such as personal computers, laptops, and mobile phones. These limitations significantly hinder their usability in real-time, on-device applications, such as home security systems requiring immediate threat detection. While training a large language model is a resource-intensive, one-time process, the operational costs of inference, the repeated running of the model to serve users, also accumulate rapidly. Industry data indicates inference makes up majority of machine learning workloads; for example, Amazon Web Services and Nvidia estimate that 80–90% of overall ML demand is devoted to inference rather than training. As LLM usage scales into millions or billions of queries, the ongoing inference energy consumption can ultimately surpass the energy cost of initial training, especially when factoring in growth in users and applications. Inference workloads not only drive-up electricity usage and data center cooling demands, but also carry significant carbon and water footprints, particularly when powered by fossil-fuel-based energy sources. This ongoing operational burden is a critical factor in the overall environmental and economic impact of LLM deployment. To address these issues, model compression techniques such as pruning, quantization, distillation, and factorization have been proposed. These methods aim to reduce model size and computational complexity without significant loss in performance. Compressed models not only reduce storage and cloud computing costs but also improve inference speed and energy efficiency. Importantly, they expand accessibility by enabling advanced AI capabilities on low-power and edge devices. These advancements are essential for building sustainable, scalable, and widely deployable AI systems.