Towards LiDAR and IMU-Based Human-Centric Scene Understanding in the Context of Urban Environments

Cities worldwide are facing increasingly complex demands driven by rapid urbanization, population growth, evolving mobility, and safety requirements. These challenges impact, for example, public and private transportation, urban planning, and public safety. Addressing them requires robust technical solutions that deeply understand the dynamics of urban environments. Urban environments are complex, dynamic scenes that are fundamentally shaped by humans acting as active agents within them, e.g., as pedestrians, drivers, or vulnerable individuals navigating city spaces. In this context, scene understanding is a core problem in machine learning and computer vision. Scene understanding refers to a system’s ability to perceive, interpret, and describe the structure and dynamics of a physical environment from sensor observations such as images, LiDAR (Light Detection and Ranging), or wearable sensor data. In urban environments, scene understanding includes recognizing pedestrians, interpreting their behavior, and reasoning about how the scene is likely to evolve over time, among others. This thesis contributes to the domain of scene understanding for urban environments with novel methods rooted in machine learning and computer vision. More precisely, it advances human-centric scene understanding in urban contexts, which means understanding dynamic scenes while explicitly placing humans and their behavior at the center of the analysis. Many approaches to human-centric scene understanding rely heavily on RGB images, which raises significant privacy concerns in public urban environments where constant visual recording is often undesirable or restricted. In contrast, LiDAR and wearable devices equipped with IMU (Inertial Measurement Unit) sensors are becoming increasingly widespread and offer a viable alternative. Unlike RGB imagery, LiDAR is inherently more robust to challenging lighting and weather conditions, while IMU sensors are completely unaffected by environmental factors. Both LiDAR and IMU are privacy-preserving by design, and combined, they provide rich structural 3D point cloud and motion information for human-centric scene understanding, without revealing visual identity cues. However, human-centric scene understanding using only LiDAR and IMU introduces several challenges: (i) Urban environments generate massive amounts of raw data far beyond what humans can feasibly annotate, limiting the use of this data for supervised learning for perception tasks such as moving object segmentation (i.e., identifying which parts of the scene are dynamic). (ii) Variations between different LiDAR sensor types or deployment setups cause domain shifts that prevent supervised models trained on one dataset from generalizing reliably to LiDAR data captured in other contexts. (iii) Multi-modal contrastive learning is, compared to the RGB domain, still underexplored for LiDAR and IMU. In the RGB domain, it has enabled joint embedding spaces between complementary modalities that greatly improve representation quality, support cross-modal retrieval, and enhance downstream tasks such as human activity recognition (HAR). (iv) Crowd simulation research has not paid much attention to learning when new agents should appear in a simulation from real-world data. It mainly focuses on realistic agent behavior policies while placing less emphasis on the combined aspect of where and when people should appear in the simulation, which is essential for realistic simulations of urban environments. In this context, this thesis presents four main contributions within the problem space arising from these challenges: First, we propose UMOS-MOTS, an unsupervised approach for LiDAR moving object segmentation (MOS) that uses self-supervised representation learning and relaxes MOS to a multivariate time series clustering problem. In stationary LiDAR settings, UMOS-MOTS achieves performance comparable to state-of-the-art supervised methods entirely without manual labels. Second, we propose LiOn-XA, an unsupervised domain adaptation (UDA) approach for LiDAR semantic segmentation (i.e., assigning each point in a LiDAR scan a semantic category). Recent methods show that UDA for LiDAR can be achieved using paired LiDAR and RGB images together with cross-modal learning and adversarial training. In contrast, LiOn-XA demonstrates that combining a 2D and 3D LiDAR representation for UDA can achieve performance comparable to existing approaches using LiDAR-RGB pairs. Third, we propose DeSPITE, a multi-modal contrastive representation learning approach that learns a joint embedding space across LiDAR, IMU, skeleton pose data, and text descriptions. This approach brings benefits of multi-modal contrastive learning into the LiDAR–IMU domain, such as enabling effective pre-training for LiDAR-based HAR and improving the interpretability of IMU signals through point cloud retrievals. Furthermore, DeSPITE demonstrates for the first time that wearable IMU signals from pedestrians can be automatically linked to their corresponding LiDAR-detected appearances, laying the groundwork for multi-modal LiDAR tasks in arbitrary multi-person scenes where such a link is not established automatically. Fourth, we propose nTPP-GMM, a combination of neural temporal point processes (nTPP) with Gaussian Mixture models (GMM) to learn the spatio-temporal patterns of when and where people appear and disappear in real scenes directly from real-world data, an aspect that has received little attention in the crowd simulation field over the past decade. The nTPP-GMM closes this gap as a learnable layer for realistic agent initialization. Together, the contributions of this thesis are building blocks for a scalable pipeline for LiDAR and IMU-based human-centric scene understanding in the context of urban environments. As a result, our contributions and findings pave the way for future human-centric scene understanding systems that respect privacy while deeply understanding how people behave in urban spaces.