Research on machine reading comprehension for BERT and its variant neural network models

Currently, machine reading comprehension models primarily rely on the LSTM networks with the gate mechanism. In the present study, we employ BERT and its variant pre-training-trained language model to conduct research and experimentation on the DuReader data set. We find that enhanced masking techniques, such as full-word mask masking and dynamic mask masking, can significantly enhance the model's performance in machine reading comprehension tasks. Therefore, the ROUGE-L and BLEU-4 values of RoBERTRoBERTa-wwm-ext model on the test set achieve ROUGE-L and BLEU-4 scores of 51.02% and 48.14%, respectively, which are 19.12% and 8.94% higher than the benchmark model. In addition, addressing the issue of suboptimal model performance is not optimal when the data is dealing with large data and the effective information is relatively dispersed. This paper employs a three-step pre-processing approach for the data set. This method is based on the F1-score to identify relevant paragraphs, answer modules and feature precomputation, so that the performance of precompute features, ultimately bringing the pre-training-trained language model, is shown to bring the model's performance closer to the average human reading comprehension level.