3D instance segmentation from multiview images

Recent advancements in technologies such as Neural Radiance Fields (NeRF) and 3D Gaussian splatting have opened new possibilities for 3D object segmentation. Research has been conducted on leveraging both appearance and geometric information for 3D segmentation. In this paper, we propose a method for 3D instance segmentation from multi-view images. Unlike conventional 2D image segmentation techniques that extract 2D regions of target objects from a single viewpoint, using multi-view images enables segmentation in 3D space by utilizing images captured from multiple perspectives of the same object. In particular, by leveraging geometric information between viewpoints, the proposed method achieves more accurate and 3D-consistent object segmentation compared to conventional 2D approaches. The proposed method optimizes 3D masks by efficiently arranging the sequence of viewpoints using camera pose information for each view. This approach is applicable not only to linear camera arrays, matrix-type arrays, and 360-degree spherical arrangements but also to general arbitrary camera placements. It can be applied to various tasks requiring ordered viewpoints, such as multi-view 3D video coding and 3D reconstruction. We applied the proposed camera arrangement method to 3D instance segmentation and conducted experiments using multi-view images containing multiple people. The results demonstrate that individual instances are effectively segmented even in cases of severe occlusion.