Traditional Chinese medicine text segmentation model with multi-metadata embedding based on Bidirectional LSTM

In recent years, natural language processing (NLP) has gained increasing attention in traditional Chinese medicine (TCM). Chinese word segmentation (CWS) is a fundamental task in NLP, focusing on accurately segmenting words based on context. However, TCM texts often exhibit incompleteness, ambiguity, and structural challenges, leading to suboptimal segmentation results. The limited size of available datasets and the inadequacy of existing models to capture the unique characteristics of TCM texts further exacerbate this issue. This paper proposes a multi-metadata embedding TCM text segmentation model (MBMC) based on a Bidirectional Long Short-Term Memory Network (BiLSTM) combined with a Multi-Head Self-Attention Mechanism (MHA). The MBMC model first constructs a comprehensive corpus of traditional Chinese medicine (TCM2023) explicitly designed for CWS tasks. This corpus fills a significant gap in the existing literature, as there were previously no dedicated datasets for TCM word segmentation. It employs Word2Vec and CNN networks to extract features from Chinese characters and their structural information, and uses them as inputs to the neural network to improve contextual feature extraction. Additionally, Conditional Random Fields (CRF) are utilized to refine segmentation outputs. On the TCM2023 dataset, the MBMC model achieved an F1-score of 95.64%, providing a reliable foundation for downstream NLP tasks in the TCM domain. Furthermore, the MBMC model obtained F1-scores of 98.93% and 97.50% on the MSRA and PKU datasets, respectively. It outperformed several pre-trained models, including BERT and Glyce-BERT, while requiring less time and memory. • Since the World Health Organization (WHO) included traditional Chinese medicine (TCM) in its latest global medical compendium, an increasing number of scholars have embarked on research related to TCM. However, TCM texts are characterized by specific linguistic contexts, and the quality of word segmentation significantly affects downstream tasks in various domains. • TCM text datasets are scarce and difficult to obtain, requiring substantial manual effort. However, high-quality datasets are indispensable for deep learning. Therefore, we constructed a custom TCM word segmentation dataset. • By analyzing the characteristics of TCM texts, we propose a multi-metadata embedding model based on BiLSTM-MHA-CRF to optimize Chinese word segmentation in TCM. By integrating state-of-the-art deep learning techniques, our model demonstrates significant performance improvements on both TCM-specific datasets and public benchmarks, highlighting its broad applicability. • Our detailed experiments validate the feasibility and effectiveness of the proposed model. Comprehensive analysis and visualization of experimental results emphasize the practical value of the model in processing TCM medical records and facilitating downstream natural language processing tasks such as entity recognition and knowledge extraction.