A novel cross-modal decoupling dual-spectral fusion system and method: A case study on the on-site estimation of fresh tobacco leaf curing characteristics

Accurate and timely estimation of easy curing characteristic (EaCC) and endurable curing characteristic (EnCC) of fresh tobacco leaves is crucial for subsequent curing and processing. This study developed a dual-spectral system for on-site estimation of EaCC and EnCC grades. The system integrated a fiber-optic spectrometer, halogen lamp, and 450 nm laser to obtain chlorophyll fluorescence (ChlF) and visible-near infrared (VNIR) spectra via mode switching. Given the unequal contributions of ChlF and VNIR spectra, a cross-modal collaborative decoupling network driven by multitasking (CMCDN-MT) model was designed to fuse ChlF and VNIR spectra for simultaneous estimation of EaCC and EnCC grades. CMCDN-MT integrated two independent subnetworks and a branch-interactive subnetwork to extract single-modal and cross-modal spectral features under the supervision of a trilateral clustering loss function. A task-driven modality fusion (TDMF) module was further designed to adaptively reorganize sub-task preference features, reducing conflicts in multitasking. Compared with single-spectral, single-task, traditional machine learning, and advanced deep learning models, the CMCDN-MT model achieved the best test performance. The dual-spectral system deployed with the CMCDN-MT model achieved 100.00 % precision in identifying low-quality EaCC and EnCC tobacco leaves, with overall accuracies of 88.00 % for EaCC grades and 86.00 % for EnCC grades, respectively. Due to the use of a multi-task strategy, the CMCDN-MT model simultaneously estimated both EaCC and EnCC grades for a sample in just 3.35 ms. Overall, the proposed method shows the potential for online applications and offers an effective approach for multi-attribute quality assessment of other industrial crops. ● ChlF and VNIR spectra are fused to estimate tobacco leaves curing characteristics. ● A novel dual-spectral cross-modal decoupling fusion framework (CMCDN-MT) is designed. ● Decoupled subnetworks collaboratively extract single-modal and cross-modal features. ● The TDMF module can adaptively reorganize and align the feature-task space. ● Dual-spectral system combined with CMCAN-MT can achieve accurate on-site estimation.