Spectral 3D computer vision: A survey of acquisition, representations, learning, and applications

Spectral 3D computer vision combines wavelength-resolved measurements with geometric information to support material-aware reconstruction, perception, and scene analysis. However, the literature remains fragmented across hyperspectral and multispectral sensing, cross-spectral depth and pose estimation, spectral point clouds, LiDAR–hyperspectral fusion, and neural scene representations. This survey provides a structured review of the field, covering spectral and geometric acquisition, data representations, calibration and registration, spectral 3D reconstruction, integrated spectral 3D acquisition and joint physical modelling, learning-based methods, downstream tasks, benchmark datasets, applications, and software tools. We organise existing methods into three complementary methodological themes: spectral support for 3D information estimation, spectral 3D reconstruction and mapping, and integrated spectral 3D acquisition and joint physical modelling. We further compare representative datasets and reported results across classification, regression, segmentation, semantic scene analysis, and spectral novel-view synthesis. The analysis indicates that spectral information can provide complementary material, condition, and visibility cues when RGB appearance or geometry alone is insufficient; however, the magnitude of the benefit is task- and baseline-dependent, and many reported gains do not isolate the spectral contribution from concurrent changes in sensing, architecture, spatial context, or fusion. Progress remains constrained by limited native spectral 3D benchmarks, inconsistent radiometric and geometric calibration, heterogeneous evaluation protocols, and limited cross-sensor and cross-domain validation. We conclude by identifying research priorities in standardised benchmarking, uncertainty-aware fusion, physically grounded learning, efficient spectral 3D representations, and generalisation across sensors, environments, and application domains.