DOROS: A multilevel traffic dataset for dynamic urban scene understanding

Abstract Advancements in autonomous vehicles and smart traffic systems require vision datasets capable of capturing complex interactions and dynamic behaviors in real‐world urban environments. Although datasets such as COCO, Cityscapes, and ROAD have advanced object detection, segmentation, and action recognition, they often treat scene elements in isolation, thereby limiting their use for comprehensive understanding. This paper presents DOROS, a dataset with multilevel annotations across Agent , Location , and Behavior categories. DOROS is designed to support compositional reasoning under diverse traffic conditions. An annotation pipeline combining foundation models with structured human refinement ensures consistent, high‐quality supervision. To support structured evaluation, we introduce the Combined mAP ( mask ) metric, which assesses instance segmentation under strict category‐level label matching while mitigating the effects of class imbalance. Extensive experiments, including ablation studies and transformer‐based baselines, validate DOROS as a resource for structured scene understanding in complex traffic scenarios. The dataset and code will be released upon publication.