Towards Relevant Human-Vehicle Interaction Data for Perceptive Machine Learning

Machine learning models, particularly those used in open-world settings like autonomous driving, require ex-tensive and diverse datasets for effective training. However, the overabundance of standard situations in large datasets often leads to underrepresentation of rare scenarios and edge cases, which are crucial for achieving high performance across all situations. This work proposes a simulation framework for capturing human-in-the-loop simulation data, enabling the creation of realistic and diverse datasets for machine learning models as well as human behavior studies. The framework combines full body motion capture with virtual reality to record scene-relevant interactions between humans and virtual objects, allowing for the generation of various ground truths in simulation. To validate the effectiveness of this approach, an action recognition model is trained on simulated data generated by the proposed framework. From the point of view of a vehicle, the model needs to determine if the vehicle is the intended target of a waving pedestrian. The investigation into different keypoint representations and spatial encodings reveals the importance of integrating spatial and contextual data for accurate action recognition. Furthermore, the results show that human-in-the-loop simulations can effectively capture complex human behaviors and interactions, enabling the creation of more realistic and diverse datasets for machine learning models.