Information Extraction-Driven Knowledge-Based Salient Masked Pre-Training Method

Closed-Book QA (CBQA) is a challenging task in the field of natural language processing. It requires the model to answer questions directly in the absence of rich contextual information and external knowledge, thus relying on the model's capacity for knowledge retention. This study proposes a salient mask optimization method based on information extraction to enhance the model's knowledge parameterization capacity in closed-book question answering tasks. Specifically, information extraction technology is first used to extract key entities and triples from text to construct salient region masks sensitive to knowledge, addressing the limitations of traditional masking strategies in professional domain knowledge coverage. Subsequently, in the intermediate pre-training stage, the salient mask is integrated with the self-attention mechanism to enhance the model's ability to learn and retain relevant knowledge. The experiment is conducted on the traditional Chinese medicine literature data from the Tianchi dataset, using T5-small as the baseline model. The experimental results show that, compared with the original model, the proposed method achieves an improvement of 11.03% in the EM score and 6.98% in the F1 score. Ablation experiments further confirm the effectiveness of information extraction in generating salient masks and highlight the role of salient masks in improving the model's knowledge parameterization capacity. This study provides a new technical solution for optimizing masking strategies in knowledge-intensive tasks and offers theoretical support for efficient knowledge compression in language models for vertical domains.