Detector‐Guided Class‐Aware Segmentation for Road User Perception in Synthetic Traffic Scenes

Precise and efficient perception of road users is essential for the safe deployment of autonomous driving systems. This study introduces a synthetic‐only perception framework for class‐aware segmentation in urban traffic scenes, leveraging recent advances in deep learning‐based segmentation models. We evaluate a range of approaches, including CNN‐based models (YOLOv8, YOLOv11), transformer‐based zero‐shot segmentation (SAM1), and a fine‐tuned variant of SAM1. While these models achieve moderate performance, they often struggle with class ambiguity and inconsistent class‐wise segmentation. To address these challenges, we propose a detector‐guided segmentation pipeline that integrates a custom‐trained YOLOv11x detector with SAM2. In this framework, YOLOv11x provides bounding box prompts that guide SAM2 to produce accurate, class‐specific masks for key road users such as cars, pedestrians, and cyclists. All models are trained and evaluated on a high‐resolution synthetic dataset simulating realistic urban conditions, enabling scalable annotation without privacy concerns. Quantitative evaluation using per‐class metrics including Dice Loss, F1‐score, precision, and recall shows that YOLOv11x offers a cost‐effective and real‐time detection backbone with fast inference and low GPU memory consumption. Furthermore, the detector‐guided SAM2 pipeline, using the SAM2.1 large model, achieves competitive performance (F1: 0.676) without any fine‐tuning, surpassing the fine‐tuned SAM1 with Vit‐H (F1: 0.671). These results highlight the effectiveness of prompt‐based segmentation when paired with a strong object detector, offering a practical, scalable solution for real‐time perception in autonomous driving systems. © 2025 Institute of Electrical Engineers of Japan and Wiley Periodicals LLC.