Augmenting Crosswalk Segmentation With Diffusion-Generated Data for Adverse Weather and Lighting Conditions

Training deep learning models for semantic segmentation requires substantial annotated datasets, which are often limited in diversity and size. This study investigates the effectiveness of using diffusion-generated synthetic data to enhance crosswalk segmentation models for assistive navigation systems serving visually impaired individuals. We hypothesized that a Stable Diffusion model fine-tuned on a small collection of 150 first-person view (FPV) images, predominantly captured in sunny conditions, could generate valuable training data representing diverse weather scenarios not present in the original dataset. To test this hypothesis, we created 1500 synthetic images with varying weather conditions using our fine-tuned Stable Diffusion model. We then trained several U-Net-based semantic segmentation models on different combinations of synthetic and original images. For evaluation, we developed a new high-quality test dataset containing 300 annotated real-world images captured across diverse weather conditions. Our results confirmed that models trained on a combination of real and synthetic images significantly outperformed those trained exclusively on the limited real-world dataset. The mixed-data model demonstrated superior generalization to weather conditions absent from the original training data, achieving higher performance metrics across all test scenarios. These findings underscore the potential of diffusion models for data augmentation when authentic datasets are constrained. The pipeline we propose offers an efficient, scalable, and adaptable approach applicable to other domains requiring synthetic data enhancement.