A Study on Short Text Classification Based on BERT-Att-BiLSTM

Aiming at the problems of sparse features, insufficient depth, and noise interference in Chinese short text, this study proposes a Chinese short text classification model that combines the BERT pretraining model, the attention mechanism, and BiLSTM (short and long term memory network). First, in the data preprocessing stage, text error correction is performed on the data to improve the accuracy of the data. Second, a BERT pretraining model is used to extract the deep feature vectors of the short text; then, an attention mechanism is introduced to assign weights to the output of BERT to highlight the key parts of the sequence; then, the attention-weighted feature vectors are inputted to the BiLSTM network to comprehensively capture the semantic information of the text; and finally, the classification results are outputted through the fully connected layer. This experiment is validated using THUCNews dataset, and the results show that the accuracy of the proposed BERT-Att-BiLSTM model reaches 94.85%, which validates the effectiveness of the algorithm.