Synthesizing Realistic Traffic Events From UAV Perspectives: A Mask-Guided Generative Approach Based on Style-Modulated Transformer

Unmanned aerial vehicles (UAVs) have emerged as valuable tools in intelligent transportation systems, offering potential for real-time traffic monitoring and emergency response. However, the limited flight endurance of UAVs restricts their ability to collect large amounts of traffic event data crucial for data-driven models. While simulation platforms provide an alternative data source, a significant visual gap persists between synthetic and real images. Given the unique characteristics of UAV perspectives, where objects typically appear smaller, and the challenges faced by existing generative approaches in maintaining structural and semantic consistency, this paper proposes a novel generative approach. Our method integrates semantic mask guidance with a style-modulated Transformer-based GAN architecture to address these issues. Our approach first utilizes the Segment-Anything Model (SAM) and Contrastive Language-Image Pre-training (CLIP) to extract semantic masks from GTA V simulated images. Subsequently, we introduce Spatially-Attentive Denormalization Modules (SADM) within the generator. These modules incorporate semantic mask statistics to enhance image quality and maintain consistency. Furthermore, we develop a perceptual discriminator incorporating a “memory bank” mechanism to more effectively evaluate image realism and stabilize the training process. To further enhance the quality of generated images, we develop a comprehensive training strategy. Our experimental results demonstrate that the proposed method outperforms existing state-of-the-art approaches in both quantitative metrics (i.e., FID and KID) and qualitative visual assessments, thus highlighting its effectiveness and superiority. Overall, our approach offers a robust solution for generating highly realistic traffic event images from UAV perspectives, effectively addressing the scarcity of real-world UAV-recorded traffic event data.