Open-SAM:Zero-Shot segmentation for incomplete 3D scenes

Recent advancements in large model-assisted 3D scene segmentation have shown great promise, but most methods still require a fully captured scene for accurate results, limiting their ability to handle unseen objects or incomplete scenes. To address this, we propose Open-SAM, an end-to-end zero-shot 3D scene segmentation framework that leverages the Segment Anything Model (SAM) for initial 2D object segmentation. The resulting high-quality 2D masks are then projected onto the corresponding 3D point cloud and used to train a 3D U-Net, effectively segmenting the seen classes. Furthermore, we introduce a Low-Rank Decomposition (LRD) module to refine feature representations and reduce redundancy, thereby increasing the model’s effectiveness even when data is incomplete or objects are unseen. To handle novel categories, we incorporate a contrastive similarity loss that separates seen and unseen classes based on their feature embeddings, enabling effective segmentation of unseen categories in new scenes. Our method distinguishes itself by maintaining high segmentation performance without full scene capture, demonstrating flexibility and effective performance in handling unseen objects and incomplete environments. Experimental results on the ScanNet dataset validate the effectiveness of our approach, as it outperforms existing methods in zero-shot 3D segmentation under various scenes.