From Failure To Fidelity: Enabling Scalable Sim2real Lidar Perception Through Realistic Digital Twins

LiDAR technology plays a key role in enabling intelligent perception across urban systems. Its ability to capture accurate 3D data regardless of lighting conditions makes it well-suited for applications in autonomous vehicles, traffic monitoring, security, and privacy-aware smart city infrastructure, offering advantages over camera-based systems in depth, precision, and reliability. However, scaling deployment faces substantial challenges due to the time, effort, and cost of gathering and labeling LiDAR data used to train and test perception algorithms. This dissertation addresses these challenges by tracing a comprehensive research trajectory from early failures to a successful, fidelity-driven solution. Initially, (1) it explores transfer learning, highlighting its limitations in generalization, and (2) develops a self-supervised method leveraging teacher-student modeling to train deep neural object detectors. Despite promising results in constrained settings, these approaches showed limited scalability and inconsistent real-world performance. In response, the dissertation studies Sim2Real learning, (3) investigating limitations, and (4) presents a novel scalable Sim2Real learning framework. This framework uses high-fidelity (HiFi) digital twins (DTs) that replicate real locations with publicly available geospatial data, preserving structural and contextual details such as background geometry, road characteristics, and traffic distributions. By simulating sensors in these HiFi DTs, in-domain LiDAR datasets are synthesized. These synthetic datasets can train perception models that consistently match or exceed the performance of models trained on real data. In addition to these methodological advances, the dissertation contributes to research through (i) LiGuard: streamlined open-source LiDAR data processing and visualization software for rapid research, (ii) UrbanTwin: open-source digital-twins of real locations advancing research beyond perception, and (iii) open-source synthetic LiDAR datasets. By systematically documenting and mitigating foundational failures, this work provides a blueprint for building robust, scalable LiDAR perception systems for real-world ITS deployment.