CaptchaSAM: Segment Anything in Text-based Captchas

While text-based captchas, designed to distinguish between human users and bots, have encountered numerous attack methods, they remain a prevalent security mechanism employed by various websites. Some deep learning-based approaches can recognize captcha character sequences end-to-end; however, the labor-intensive and time-consuming labeling process severely restricts their feasibility. In this study, we introduce CaptchaSAM, to segment anything in text-based captchas. Our insight lies in the fact that identifying individual characters is a simpler task compared to recognizing character sequences, leading to a substantial reduction in labeling dependency. To accomplish this, we utilize the Segment Anything Model (SAM) for character-level semi-automatic annotation. Subsequently, we leverage the annotated data to train a semantic segmentation model. Our experiments with real-world captcha systems demonstrate that CaptchaSAM significantly outperforms state-of-the-art methods with just a few labeled captchas. We anticipate that our research will encourage security experts to reconsider the design and deployment of text-based captchas. The source code is accessible at https://github.com/SJTU-dxw/CaptchaSAM.