Hyperspectral images 3D reconstruction based on structure-from-motion and multi-view stereo

• An integrated segmentation-first framework is proposed to suppress background noise and edge artifacts in hyperspectral 3D object reconstruction. • A custom matching strategy for circular scanning reduces the influence of repetitive textures in camera calibration, and a cost function avoids ambiguous correspondences in stereo matching. • To generate hyperspectral point clouds consistent with uniform illumination, a multi-view fusion mechanism is introduced. • Experimental evaluation using the reprojection contrast metrics demonstrates the high geometric and spectral quality of the reconstructed hyperspectral point clouds. Hyperspectral 3D imaging technology aims to obtain 3D models with hyperspectral reflectance textures, holds significant potential in fields such as medical imaging, precision agriculture, and optical sorting. However, the generation and application of hyperspectral 3D models face numerous challenges due to the complexities of light propagation in intricate environments, particularly in interactions with various media. In order to achieve a one-to-one mapping between hyperspectral data and point cloud coordinates, many researchers have developed different hyperspectral three-dimensional imaging systems to address the problem. In this paper, we propose a framework based on the Structure-from-Motion (SFM) and Multi-View Stereo (MVS) to process the multi-view hyperspectral images collected by an internal push-broom hyperspectral camera. In the framework, an image segmentation module is introduced to suppress the background noise during the target reconstruction process. In the Structure-from-Motion algorithm, a matching strategy for circular scanning mode is adopted to effectively alleviate the influence of repetitive textures and large-span mismatches on camera pose estimation, thereby improving the pointcloud stitching accuracy. And we utilize the multi-channel redundant information to enhance the robustness of stereo matching depth estimation. Additionally, we introduce a fusion mechanism that incorporates mask-based noise suppression and statistical reflectance integration to enhance geometric accuracy and spectral fidelity. The hyperspectral reflectance is not severely distorted through aggregating the spectrum of redundant observed points. Experimental results on real-world data collected by our system demonstrate that the effectiveness of our method to handle spectral variability and geometric complexity.