Trends in Scaling Laws for Large Language Models

This literature review synthesises a decade of research on the evolution and impact of scaling laws in large language models. The analysis highlights clear trends in increasing model size, dataset volume, and computational resources, demonstrating how these factors contribute to improvements in model efficiency and performance. It identifies a direct relationship between greater resource investment and enhanced results up to a point, after which performance gains begin to diminish. The review further reveals that while larger models generally excel across many natural language processing tasks, scaling benefits vary by task and sometimes plateau or decline. Economic considerations are also prominent, with training costs rising sharply relative to incremental performance improvements, emphasising the need to find optimal scaling points that balance cost and quality. Importantly, this review draws upon and consolidates findings from existing studies rather than presenting original empirical research. Finally, it discusses emerging challenges and potential innovative training methods based on recent scholarship. These insights offer valuable guidance for researchers and industry professionals seeking to effectively develop and apply large language models.