Improving the Autonomous Vehicle Vision with Synthetic Data Using Gen AI

This research focuses on addressing the challenges associated with training object detection and segmentation models for autonomous vehicles using real-world data, which is often difficult to collect, limited in diversity, and expensive to annotate. To overcome these limitations, we propose a novel framework for generating synthetic datasets using deep learning techniques and generative AI. Specifically, we utilize the Stable Diffusion models to generate synthetic images, and annotate them using a combination of the Segment Anything Model (SAM) and the CV AT tool. The annotations are further refined to produce label files, and the dataset is split into training and validation sets. For evaluation, the YOLOv8 is applied for object detection and segmentation tasks using the synthetic dataset. Additionally, the Salesforce/Blip image-captioning model is integrated to provide descriptive captions for the detected objects. Experimental results demonstrate that synthetic datasets generated by this approach not only provide coverage of complex and rare scenarios but also achieve comparable or superior performance compared to real-world datasets in object detection and segmentation tasks. These findings underscore the potential of synthetic data to serve as a cost-effective and scalable alternative to real-world data for autonomous vehicle training.