A multi-channel modal fusion hypergraph convolutional neural network for Tibetan medicinal material classification

The key to the digital and intelligent development of Tibetan medicine lies in the accurate identification of Tibetan medicinal plants. To address the challenges in Tibetan medicine identification—such as the susceptibility of small image targets to occlusion, interference from strong light noise, over-smoothing in bi-modal fusion, and insufficient fine-grained features in hypergraph construction—this study proposes MHCNet, a multi-channel modal fusion hypergraph neural network model for the classification of Tibetan medicinal plants. Based on bi-modal data of Tibetan medicine images and texts, the model first improves upon the DenseNet121 architecture to initially mitigate the issues of small target occlusion and strong light noise, thereby extracting robust visual features. Concurrently, a pre-trained Chinese BERT model is employed to extract the contextual semantic features of the Tibetan medicine texts. Subsequently, the extracted image and text features are fused at the feature level. The Manhattan distance formula is utilized to calculate the feature similarities across image, text, and the fused modalities for each sample, which in turn are used to construct hypergraphs for the three modalities. Furthermore, a multi-channel hypergraph neural network is designed to conduct collaborative learning on these three modal hypergraphs, effectively addressing the over-smoothing issue inherent in bi-modal fusion. Finally, a feed-forward network is introduced after the first hypergraph convolution layer to iteratively optimize and dynamically adjust the reconstruction of the hypergraph structure, resolving the inadequacy of fine-grained features during hypergraph construction. Experimental results on a self-built Tibetan medicine image-text dataset demonstrate that the proposed MHCNet model achieves a recognition accuracy of 98.11%. It effectively captures high-order cross-modal correlations, providing a reliable solution for the intelligent identification of Tibetan medicines.