Automating synthetic dataset generation for image-based 3D detection: a literature review

Abstract Reliable 3D detection is fundamental to autonomous systems such as mobile robots, self-driving cars, and unmanned aerial vehicles (UAVs). To achieve this capability, researchers have developed and trained supervised networks, which require large amounts of diverse and precisely annotated data. Due to the complex, expensive, and time-consuming capturing and annotation process, synthetic dataset generation approaches have gained popularity over the last decade. With increasing computational resources and advances in simulation technologies, a variety of dataset generators have emerged. These methods rely on either traditional 3D modeling or neural image synthesis to generate data for specific scenarios or general-purpose 3D detection tasks. Their primary goal is to produce high-quality, annotated 3D datasets in an automated and scalable manner. In this review, we evaluate the extent to which state-of-the-art approaches fulfill this goal by introducing a categorization scheme and conducting a comprehensive analysis of both 3D modeling and neural synthesis methods. Our analysis includes techniques used to address the Sim-to-Real domain gap. Furthermore, we assess each method’s level of automation, prerequisites, and practical adoption. This review aims to guide the reader in selecting automated dataset generation workflows for specific detection problems. By considering dataset quality, prerequisites, and application scenarios, we offer practical insights into identifying suitable methods for diverse downstream tasks.