A Review of the Current State of Multimodal Data Annotation in Traditional Chinese Medicine

Integrating artificial intelligence with traditional Chinese medicine (TCM) is a vital step towards the modernization and intelligent development of TCM. The aim of this article is to explore the current state of research and development in the annotation of multimodal data for TCM. First, the paper elucidates the background and core significance of multimodal data annotation. Then, it provides a systematic analysis of the various sources of TCM data, including textual materials such as classical texts and modern medical case records; imaging data such as facial, tongue, and ocular diagnoses; and signal data such as pulse and auscultation diagnoses. It also covers the technical characteristics of the annotation of these data types. Subsequently, the article explores detailed annotation methodologies tailored to various application scenarios. These include knowledge representation for classical Chinese medicine texts, in-depth annotation to capture the expertise of renowned traditional Chinese medicine practitioners, and standardized annotation for modern clinical information systems. Finally, this paper focuses on exploring the cross-disciplinary integration of global workspace theory (GWT) in cognitive neuroscience with multi-granularity computing and multimodal annotation in computer science. GWT offers an approach to understanding how diverse diagnostic information is synthesized into a coherent awareness, mirroring the process of TCM pattern differentiation. Multi-granularity computation is a methodological approach to processing this information through attention mechanisms at different levels. By integrating these advanced concepts with a practical framework for multimodal annotation, this paper aims to provide theoretical support and methodological insights for constructing intelligent diagnostic and therapeutic systems tailored to the cognitive characteristics of TCM.