NeRFing the Road: Synthesizing Simulated and Real-world Data for 3D Reconstruction of Large-Scale Autonomous Driving

Neural Radiance Fields (NeRF) have emerged as a promising research avenue in Autonomous Driving (AD), delivering photorealism in synthesizing novel views and 3D reconstruction. However, training NeRFs require multi-view captured datasets with accurate camera poses. The scarcity and expense of obtaining real data have made synthetic data a compelling alternative. Thus, this study has utilized synthetic data generated by the CARLA simulator to train NeRFs in various driving environments, identifying optimal configurations for high-quality view synthesis. This approach provides technical and cost-saving advantages, minimizes the need for extensive trial-and-error processes in real-world data collection, and thus promotes sustainability. The identified configurations are subsequently applied to real-world data, enabling the training and evaluation of autonomous driving (AD) agents. The presented study has established a seamless connection between CARLA and Nerfstudio and has conducted experiments in four configurations, outlining various factors such as area coverage, data capture pipeline, image resolution, density of dataset, and large-scale area coverage with state-of-the-art NeRF models designed for complex driving scenarios. Besides the standard evaluation process of NeRFs, two additional video quality metrics VMAF and FovVideoVDP are included to ensure the reconstructed views look visually accurate while maintaining perceptual quality across various viewpoints. The results show significant improvements in NeRF performance, such as an increase in PSNR from 23.37 to 26.94, SSIM from 0.72 to 0.9, and LPIPS from 0.42 to 0.12, highlighting the potential of large-scale NeRF models for accurately reconstructing complex and large-scale driving scenarios. Source code available at https://github.com/shairatabassum/carlo.