CVK-Net: A non-destructive identification method for black tea fermentation stages using multi-source data fusion

To address the challenges of accurately assessing the fermentation degree of black tea using traditional manual methods and issues such as sample destruction and low accuracy associated with other methods, we propose a model named CVK-Net, which integrates a Convolutional Neural Network (CNN), Vision Transformer (ViT), and Kolmogorov-Arnold Network (KAN). By obtaining sensor data and extracting RGB, HSV, and Lab color components from images, numerical data is constructed and aligned with image data to build a bimodal dataset. First, dynamic snake convolution was introduced to improve the dense connection block for extracting shallow image features, while KAN was introduced to extract numerical modal features. The ESE attention mechanism was then used to generate bimodal feature weights and achieved adaptive fusion. Finally, ViT was used to construct a bimodal global feature representation. In addition, we developed a black tea fermentation identification system. The experimental results show that the model achieves an accuracy of 98.15 %, a precision of 98.20 %, a recall rate of 98.30 %, and an F1-score of 98.11 %, demonstrating good identification performance. CVK-Net can accurately identify the fermentation stages of black tea, providing effective technical support for non-destructive testing during the black tea fermentation process. • A superior black tea fermentation identification system has been developed. • A bimodal dataset with images and sensor numerical data was constructed. • The proposed CVK-Net model provides an impressive accuracy of 98.15 %. • Kolmogorov-Arnold Network efficiently extracts features from numerical data. • Dynamic snake convolution and attention mechanisms enhance feature extraction.