Corpus Classification Algorithm Based on BERT Pre-trained Model

Among the many tasks of NLP, corpus classification is a basic task, which is widely used in the fields of text classification, sentiment analysis and information retrieval. BERT’s core innovation is the bidirectional encoder, which makes use of contextual information at the same time to understand the semantics of text more fully. Based on the fine-tuning of Sentence, BERT can achieve the state-of-art effect in almost all NLP tasks, significantly reduce the training cost, and overcome the limitation of BERT in sentence level representation. The data model in this study is composed of AM, BERT embedding layer, sentence-based fine-tuning and BERT pre-training method. The verification experiment consists of model selection, data set selection, parameter optimization scheme, model training results and comparison with baseline model. By improving and optimizing the BERT model, this study enhances the model’s ability to understand text semantics, and lays a solid theoretical foundation for the fine-tuning of downstream corpus classification tasks.