An Efficient Encoding Spectral Information in Hyperspectral Images for Transfer Learning of Mask R-CNN for Instance Segmentation of Tomato Sepals

The most vulnerable parts of tomatoes are the tips of the sepals, which are the primary entry points for fungal spores. Their precise segmentation within hyperspectral images (HSIs) plays a pivotal role in the development of automated and non-destructive systems for assessing tomatoes’ sensitivity to fungal infections. This research addresses the critical need for encoding spectral information in hyperspectral imaging to enhance the efficiency of such automated systems. We investigate four different techniques: Principal Component Analysis (PCA), Independent Component Analysis (ICA), Probabilistic Principal Component Analysis (PPCA), and Non-Negative Matrix Factorization (NMF), to perform transfer learning for tomato sepal instance segmentation using models previously trained on RGB images. A comparative analysis of three Mask Region-based Convolutional Neural Network (Mask R-CNN) backbone models is conducted: the Faster R-CNN, Deformable ConvNet, and Feature Pyramid Network (FPN) on spectral-encoded HSIs of the Brioso tomato variety. The Mask R-CNN with FPN, integrated with the NMF technique achieved the highest level of accuracy, yielding a Mean Average Precision (mAP) of 94.05%. Furthermore, on the second dataset, which included an additional three tomato varieties: Capricia, Provine, and Sao Paolo, the same model achieved mAP score of 86.42% across all tomato varieties with only a single false positive detection. Additionally, we incorporated a custom convolutional layer initialized it with estimated NMF coefficients, and achieved a mAP score of 87.40%. This demonstrates the potential of integrating spectral information encoding with trained deep learning-based instance segmentation models to enable robust and accurate transfer learning for automated agricultural food quality assessments.