TDiff-HSI: Tucker-guided diffusion for high-dimensional RGB-to-HSI image generation

Abstract We introduce TDiff-HSI, a diffusion-based model that can generate hyperspectral images (HSIs) directly from RGB images and material-wise segmentation masks. HSI provides both spatial (u, v) and spectral (λ) information. The accompanying dataset that we are releasing spans wavelengths in the range from 420 to 1728 nm, digitized into 512 channels. Directly handling this immense three-dimensional dataset is computationally prohibitive and often leads to numerical errors. To address this challenge, TDiff-HSI leverages Tucker decomposition to reduce dimensionality, enabling more stable and efficient processing. Moreover, spectral precision is enhanced by combining RGB channels with a material segmentation mask. To support this research, we constructed a new dataset using a hyperspectral camera. The dataset comprises 40 014 RGB-HSI pairs across 78 scenes, featuring 12 objects with corresponding polygonal segmentation labels. Experimental evaluation demonstrates that TDiff-HSI achieves state-of-the-art performance verified on the existing dataset. For the new dataset that we are releasing, we establish new benchmarks of MRAE 0.2169, RMSE 0.0192, PSNR 36.46 dB, SAM 0.0424, and SSIM 0.9327 Project and dataset are available at https://github.com/JaeikBae/TDiff-HSI