3D instance segmentation by optimal view sequence ordering

In this paper, we propose a geometrically optimized method for 3D instance segmentation of complex target objects from multi-view images. Unlike previous studies that perform optimization based on a random order of input viewpoints, our method determines the optimal viewpoint order to maximize the 3D information of the target object and minimize information loss due to viewpoint changes during successive optimizations. This method is specifically designed to address severe occlusion and cross-view identity mismatch issues. To achieve this, we present an fully automated 3D segmentation method by integrating a novel SE(3) distance-based viewpoint sequencing strategy and a robust cross-view 3D instance matching algorithm. Optimal view sequence ordering minimizes the geometric distance between camera poses, maximizing viewpoint continuity and improving optimization convergence. Geometric instance matching ensures identity retention even under occlusion conditions by considering the 3D structure of the target object. In this study, we generate a 3D consistent instance mask that can be applied to any new viewpoint. Experimental evaluations on public datasets (Mip-NeRF 360, LERF, Tanks, and Temples) and custom datasets (people1, people2) show that our method outperforms existing 2D and 3D instance segmentation methods, achieving higher mIoU scores and more complete segmentation under occlusion.