Instance Segmentation of Open Vocabulary Images by Fusing Fast Fourier Transform

Open vocabulary instance segmentation is an important research direction in the field of computer vision, where open vocabulary instance segmentation utilizes natural language descriptions to recognize and segment never-before-seen objects in an image, as opposed to traditional instance segmentation that relies on a fixed set of categories. This approach, by building a bridge between visual and linguistic features, allows the model to understand and segment new object categories through language without relying on additional human annotations. Thus, open vocabulary instance segmentation has the potential for a wide range of applications by enabling models to understand and segment unknown categories of objects. However, the current open-vocabulary instance segmentation is still unsatisfactory for neighboring, overlapping objects and irregularly shaped objects. The difficulty of the model's segmentation for neighboring and overlapping objects mainly comes from the blurring of the boundaries between the objects, whereas the model may have difficulty in accurately capturing the details of these shapes when confronted with irregularly shaped objects, resulting in poor segmentation accuracy. Therefore, we propose IVFF: an open vocabulary image instance segmentation method fusing Fast Fourier Transform, which enhances the edge features and detail reconstruction of the image by introducing Fast Fourier Transform to process the frequency domain information, improves the differentiation between neighboring and overlapping objects in the image, and enhances the instance segmentation effect on irregular objects. We validate it on the COCO dataset, and the enhancement of AP metrics proves that this method can effectively improve the segmentation effect.