Accelerating Large Language Model Pretraining via LFR Pedagogy: Learn, Focus, and Review

We introduce an effective and scalable data selection technique to accelerate the pretraining of large language models (LLMs).Given the variation in quality and informativeness of web-scale corpora, we present the Learn-Focus-Review (LFR) paradigm-a dynamic training approach that adapts to the model's learning progress.Inspired by human learning techniques like spaced repetition, LFR tracks the model's learning performance across data instances and prioritizes revisiting challenging and diverse regions of the dataset that are more prone to being forgotten, enabling better retention and more efficient learning.Through experiments spanning over 2200 GPU hours, we show that LFR significantly enhances data efficiency in pretraining while improving downstream performance across commonsense reasoning, question answering, problemsolving, language modeling, and translation tasks.LFR consistently achieves lower perplexity and higher accuracy using just 5%-19% of the training tokens as models trained on the full dataset.Notably, LFR matches the performance of industry-standard Pythia models with up to 2 the parameter count while requiring only 3.2% of the training tokens.