Mold spot detection for paper artifacts based on multimodal feature fusion

Chinese paper-based cultural artifacts are threatened by mold deterioration, necessitating precise mold identification for effective conservation. This study introduces a hyperspectral imaging-based triplePath multimodal feature fusion network (TPMFN) to improve the sensitivity and timeliness of traditional methods. It combines spatial and spectral features using an advanced 2-D CNN with spatial attention for RGB details, a SpectralFormer for analyzing 400–1000 nm hyperspectral data, and a hybridCNN for extracting spectral-spatial features. Various cross-modal fusions (concatenation, addition, multiplication) facilitate efficient integration and classification of multi-source data. Testing six mold types on paper artifacts, TPMFN significantly outperforms SVM, 1-D CNN, 2-D CNN, SSFTT, HybridSN, and SpectralFormer on OA (98.78%), AA (98.52%), and Kappa (98.21%), with a 3.83% improvement in detail recognition. Ablation studies confirmed a 7.5% feature discriminability enhancement from the dual attention mechanism. This research provides a high-precision, practical solution for paper artifact mildew detection, crucial for establishing preventative conservation.