Segment Anything in Light Fields for Real-Time Applications via Constrained Prompting

Segmented light field images can serve as a powerful representation in many computer vision tasks exploiting geometry and appearance of objects, such as object pose tracking. For those images, segmentation presents an additional objective of recognizing the same segment through all the views. Segment Anything Model 2 (SAM 2) allows for producing semantically meaningful segments for monocular images and videos. Using the video SAM 2 on less general 4D functions such as light fields is ineffective. In this work, we present a novel segmentation method that adapts SAM 2 to the light field domain without retraining or modifying the model. By utilizing epipo-lar constraints, our method produces high quality and view-consistent masks, outperforming the SAM 2 video tracking baseline and working 7 times faster, moving towards a real-time segmentation speed. We achieve this by exploiting the epipolar geometry cues to propagate the masks between the views, probing the SAM 2 latent space to estimate their occlusion, and further prompting SAM 2 for their refinement. The code and additional materials are available at https://roboticimaging.org/Projects/LFSAM/.