Augmenting Pretrained Language Models with N-gram Embeddings through Incremental Training
As a new scaling technique for large language models, the n-gram embedding improves the base ability of a model while adding almost no inference cost. Mainstream research realizes it through joint training of the n-gram embedding with the backbone, which severely restricts the settings in which an n-gram embedding can be used. We therefore propose training an n-gram embedding onto an existing model through incremental training, to extend where the technique can deliver value. Specifically, we explore several settings of the n-gram embedding under incremental training---the learning rate of the embedding, the vocabulary size, and the injection position---and train an n-gram embedding onto the existing model under the best configuration. Experiments show that the best configuration under incremental training does not fully coincide with that under pretraining; moreover, the incrementally trained knowledge is written predominantly into the n-gram embedding, while the general ability of the backbone is retained throughout this training process.
Authors
- Caoyun Fan (ORCID: https://orcid.org/0000-0001-7360-3468)
- Fangming Li
- Minghao Shao (ORCID: https://orcid.org/0009-0002-4467-6224)
- Haofei Yang
- Nanhui Zhao
- Shaohua Hu
- Xusheng Gu
Publication Details
- Journal
- arXiv (Cornell University)
- Published
- 2026-11-16
- DOI
- https://doi.org/10.48550/arxiv.2611.20415
- Primary Topic
- Topic Modeling
- Type
- preprint