Efficient Language Models through Static Vocabulary Pruning of Low-Impact Tokens

Recent advances in language models have led to increasingly large vocabularies, many of which contain tokens that contribute minimally to model performance while significantly increasing computational overhead. We present a static vocabulary pruning technique for language models that removes low-impact tokens based on their contribution to the loss function. Our method reduces model size by 15% and increases inference speed by 20% while maintaining comparable perplexity across three benchmark datasets. Statistical analysis confirms these improvements across multiple runs, suggesting vocabulary pruning offers a promising direction for optimizing language models in resource-constrained environments.