Boosting Vision Language Segmentation via Pseudo-Report Generation in Weakly Paired Stroke Datasets

Motivation: Vision Language Segmentation Models (VLSMs) have excelled in medical image segmentation by leveraging both image and text data, improving clinical support. However, limited access to fully paired image-report-mask datasets restricts their effectiveness. Goal(s): We aim to boost VLSM performance by converting weakly paired datasets with limited reports into fully paired datasets using pseudo-report generation without additional training. Approach: Using a Cross-modal Self-Retriever, we generated pseudo-reports with a pre-trained Vision-Language Model and trained VLSMs with these reports and images. Results: Our method notably improved segmentation performance, outperforming image-only models on the DSC metric with as little as 10% of data containing reports. Impact: Our pseudo-report generation approach maximizes VLSM potential in report-limited environments without additional training, enhancing efficiency. Notably, with only 10% of reports available, it outperforms image-only models and more effectively reduces false positives, providing practical clinical benefits.