An image captioning framework for therapeutic description generation of Traditional Chinese Medicine herbs

Generating therapeutic descriptions for Traditional Chinese Medicine (TCM) herbs from images is challenging due to subtle inter-class visual variations, domain-specific knowledge complexity, and the lack of large-scale benchmark datasets. Therapeutic descriptions are essential in TCM as they provide information on the functions, properties, and clinical applications of herbs, supporting identification, education, and clinical decision-making. To the best of our knowledge, this is the first study to formulate TCM therapeutic description generation as an image captioning task. We propose a novel image captioning framework, UVA-Cap, which introduces an improved attention mechanism called Upgraded Visual Attention (UVA). UVA enriches conventional attention by fusing global visual context with localized feature selection, resulting in a more comprehensive and context-sensitive visual representation that significantly improves the quality, coherence, and semantic consistency of generated captions. In addition to introducing the novel model UVA-Cap, the framework employs a ResNet-50-based CNN to extract both global and local visual features and evaluates multiple captioning architectures, including Single LSTM variants, Dual-LSTM models, and the Up-Down model. To support this task, we construct the TCM-TheraCap dataset with 130,381 images of 80 TCM herb species, each paired with detailed therapeutic annotations. Experimental results show that the proposed approach generates accurate and semantically meaningful therapeutic descriptions, establishing a benchmark for future research in TCM herb understanding.