SpecColorNet: An interpretable multimodal deep learning approach for predicting SSC of multiple pears

Visible/near-infrared spectroscopy provides a non-destructive approach for evaluating soluble solids content (SSC) of pears ( Pyrus pyrifolia Nakai). However, variations among pear cultivars, especially in pear color, markedly affect spectral reflectance, thereby limiting the cross-cultivar SSC prediction. To address this, we developed SpecColorNet, a multimodal deep learning framework that integrates spectral and peel color data to accurately predict the SSC of six different pear cultivars simultaneously, and interpreted its decision-making mechanism with gradient-weighted class activation mapping++ (Grad-CAM++). The prediction accuracy of the multi-cultivar SpecColorNet was improved by 9.42 % compared to the multi-cultivar model based solely on spectral data with corresponding RMSEP values of 0.63, 0.60, 0.92, 0.70, 0.54 and 0.55°Brix for six different pear cultivars. In addition, the prediction accuracy of SpecColorNet was comparable to that of single-cultivar spectral models. The interpretation analysis indicated that the SpecColorNet successfully directed the model's attention to the 555–640 nm spectral region, where the most marked cross-cultivar differences occur, thereby improving its generalization and accuracy. Overall, this study proposed a multimodal approach for robust SSC prediction across diverse pear cultivars, which overcame the challenge of cultivar diversity and peel color variability to enable multi-cultivar models with enhanced robustness over pure spectroscopy-based methods. • Develop a SpecColorNet model to fuse spectral and peel color data information. • The multi-cultivar SpecColorNet model matches single-cultivar spectral model performance. • Interpret the learning mechanism of the SpecColorNet model through feature visualization.