A Method for Discovering Ancient Chinese Neologisms Based on the Combination of Information Entropy - Mutual Information and BI-LSTM-CRF Model

Traditional new word discovery often relies on the information entropy-mutual information (IM) algorithm. In this study, we propose a joint IM algorithm and bidirectional long short-term memory conditional random field (BI-LSTM-CRF) neural network model, referred to as IM-BI-LSTM-CRF. This model retains the advantages of the IM algorithm while leveraging the BI-LSTM model to capture contextual dependencies and the CRF layer to model label dependencies. Using this hybrid approach, we conducted new word discovery experiments on a self-annotated dataset, utilizing word embedding vectors and different word position annotation sets. The results demonstrate that the IM-BI-LSTM-CRF model surpasses traditional LSTM and BI-LSTM models in new word discovery and exhibits strong generalization capabilities.