Hyperspectral neural radiance fields for 3D scene understanding of gas plumes

Hyperspectral images (HSI) can be used in material detection and identification, having applications ranging from environmental monitoring to national security. Specifically, longwave infrared (LWIR) HSI is commonly used to detect and identify gas plumes. Existing plume detection methodology is based on single view, often nadir images of a plume, resulting in limited understanding of plume geometry and background context. Capturing images of a gas plume from multiple angles can potentially increase detection and identification performance, as well as enable 3D scene understanding and plume reconstruction. The recent advent of neural radiance fields (NeRFs) represents a paradigm shift in 3D scene understanding and novel view synthesis, implicitly encoding volumetric scene properties within a neural network. We explore the use of NeRFs to create a latent 3D scene understanding of LWIR HSI containing a gas plume, and the possibility of applying such methodology to downstream tasks, specifically gas plume detection, to novel rendered views. This is enabled by high fidelity simulated HSI scenes from the physics-based DIRSIG software suite. We adapt the Mip-NeRF architecture for use with HSI and demonstrate its ability to reconstruct novel view HSI. Gas plume detection performance is demonstrated on the novel views, and a study on detection performance as a function of the number of images is conducted.