Masked Structure-from-Motion: Novel View Synthesis in Undersea Operations

Neural radiance fields (NeRFs) have proven effective at generating models for novel view synthesis from a collection of images captured at various perspectives around buildings, statues or other structures. Mobile robots could exploit these NeRF models in many ways, including rehearsing proposed path plans, validating in situ sensor measurements from the experiences of previous sorties into an area, incrementally building a model from a series of partial observations, or infilling temporarily occluded sections of a live observation. Maritime operations bring several challenges to the NeRF training, as the captured imagery is often impacted by water effects - including splashing and droplet formation, through-water optical clarity and the independent motion of various floating objects. In this work, we address building NeRF models in maritime applications.In most prior NeRF work, static scenes are assumed, leading to artifacts and distortions in the reconstructed views when changes occur during the data capture. However, many real-world scenes involve dynamic elements, such as people, vehicles, or animals, which cannot be ignored. In some cases these dynamic elements are the intended reconstruction targets of the novel view synthesis. In other cases the dynamic elements are interferers that obstruct the view of the underlying scene. We develop a framework for generating effective NeRF models for both of these scenarios that is generalizable to various scenes, aiming for applications to autonomous vehicles in the maritime domain.We leverage foundation models for detection and image segmentation in an automated pipeline to provide curated data for effective training of NeRF models in these applications. By masking out the tracked objects, we reveal the underlying static background. Alternatively, a tracked object may be masked in to create a novel view synthesis in a moving frame of reference attached to the object. Our approach enables NeRFs to capture the complexity and dynamics of real-world scenes, enhancing the accuracy and realism of the reconstructed views.We demonstrate our techniques on underwater imagery of a coral reef occluded by colorful fish, and above-water imagery taken from a submersible robot with a camera affixed to a small mast. Our proposed approach outperforms models generated using unsupervised structure-from-motion pose generation and direct use of camera pose from onboard navigation in both cases, based on widely used video quality metrics.