A collaborative framework for memory-augmented video object segmentation

We introduce SAM2-Cutie, an innovative video object segmentation (VOS) network that incorporates segmentation quality assessment without requiring ground truth annotations, thereby enhancing the accuracy of Cutie through the strengths of the state-of-the-art instance segmentation models SAM2(Segment Anything 2). Unlike traditional semi-supervised VOS approaches that perform segmentation in a single step without correction, SAM2-Cutie adopts a Collaborative process inspired by interactive VOS methodologies. Our network not only evaluates the quality of video object segmentation in the absence of ground truth but also automatically corrects hard frames, ensuring higher segmentation accuracy and reliability. Particularly, our model has demonstrated superior performance on the long-term video dataset, on LVOS-val, SAM2-Cutie improves by 1.1 𝒥 and ℱ over Cutie.