Low-effort Iterative Dataset Generation Pipeline for Unknown Object Instance Segmentation

Robots operating in everyday environments encounter a wide variety of previously unseen objects. Deep Learning methods simplify unknown object and scene segmentation by structuring inherent real-world complexities, improving visual scene understanding. However, they need vast amounts of labeled high-variance data for training. Acquiring these labels for rich real-world data requires significant manual effort, especially for segmentation masks. Although interactive segmentation accelerates this process, these methods still require substantial manual interaction, and the creation of large datasets remains labor-intensive. Consequently, there is a lack of diverse, high-quality datasets for unknown object instance segmentation in everyday environments. This research proposes a semi-automatic, RGB-only algorithmic pipeline for annotating novel objects, reducing manual effort to iteratively placing objects in the scene. We investigate several change detection-based approaches, including remote sensing change detection methods (TTP model), the DeepBackgroundMattingV2 image matting model, and the Segment Anything Model (SAM1 + SAM2) prompted with automatically extracted change regions. We propose the novel ILIS dataset to evaluate these methods in challenging everyday scenes, displaying reliable automatic mask proposal performance of up to 0.9549 mIoU and 0.9565 boundary F1 score. This highlights the potential of this method to accelerate large-scale dataset creation, saving at least 27.27 hours per 1,000 images by eliminating manual annotations.