Materials-aware Entity Augmentation for Continued Pre-training in Materials Science

While NLP-based approaches have shown promise in accelerating materials science research, scarcity of training data containing materials-aware entities hinders the development of models that generalize across domain tasks. To address this challenge, we present MENTAP, a novel continued pre-training framework that employs preliminary data augmentation via materials-aware entity extraction and differential masking strategies. By improving the representation of rare entities in training data, MENTAP effectively mitigates long-tail distribution challenges, enhancing the frequency and diversity of materials-aware entities. Comprehensive evaluations demonstrate that MENTAP outperforms existing continued pre-training methods across benchmarks, underscoring its effectiveness in adapting PLMs to materials science.