Semantic Image Segmentation Annotation Refinement Using SAM

With the current emergence of increasingly large AI models, the amount of data needed for their training is growing rapidly. In many cases, such as in computer vision, the creation of suitable datasets is associated with high costs due to the laborious annotation process. Recently, a growing number of foundation models have emerged that can be used for a variety of tasks. One example is the Segment Anything Model (SAM), which enables interactive object segmentation in images and can thus be used during the creation of datasets, speeding up the annotation process tremendously. In this paper, an approach is proposed that allows existing coarse annotations to be refined in a fully automated way using SAM. It enables existing datasets to be enlarged efficiently. The proposed approach is demonstrated using the Cityscapes dataset, which contains 2975 finely annotated training images in addition to 20000 coarsely annotated images, thus allowing the amount of training data to be significantly increased through refinement. The extensive evaluation performed examines the influence of various hyper-parameters and demonstrates the effectiveness of the approach both quantitatively and qualitatively.