Real-time spectral reconstruction method with noise robustness based on a 2D-CNN network for snapshot spectral imaging sensors

The capability of simultaneously capturing spatial and spectral information represents the most distinctive advantage of spectral imaging technology. To expand its applicability to different scenarios, compact and cost-effective spectral imaging systems are necessary. The mosaic spectral imaging sensor is considered a promising technical pathway for future consumer-grade video spectral imaging, as it offers snapshot acquisition along with advantages such as compactness and integration flexibility. However, this technology heavily relies on spectral reconstruction algorithms, many of which suffer from either excessive computational latency or insufficient robustness to noise, so that their practical utility is limited. In this study, we developed a noise calibration protocol for mosaic spectral sensors based on the sensor noise theory as well as a high-fidelity DN simulation algorithm. These efforts enable the efficient construction of a massive and realistic DN-spectrum dataset. Based on this dataset, we further proposed a spectral reconstruction model using a two-dimensional convolutional neural network (2D-CNN). As a result, the newly developed model demonstrates a strong noise robustness and balanced capabilities in reconstructing the samples with substantial high-level noise, and thus achieves an average reconstruction fidelity of 98.37%, which maintains a level of performance that is comparable to or exceeds that of high-fidelity and complex traditional algorithms. The reconstruction RMSE ratio is only 1.87%, which is only half of that of traditional algorithms. Moreover, the full-image reconstruction time is less than 1 second, which is significantly faster than current high-fidelity algorithms that often require tens of minutes. We believe these advancements will enable lightweight spectral imaging systems to achieve accurate real-time spectral imaging in wide scenarios or under harsher lighting conditions.