Contextualizing the Sim-to-Real Gap for Hardware-in-the-Loop Testing of Highly Automated Driving Functions

Real-time closed-loop test systems, such as hardware-in-the-loop, are state-of-the-art tools for the verification and validation of highly automated driving functions. As the functional scope of driving functions expands, it becomes necessary for automated systems to perceive their environment accurately and integrate seamlessly into the existing traffic. To achieve this, vehicles are incorporating machine learning components, ranging from environmental perception and driving policies to trajectory planning and control. Given that these subsystems are designed to operate in real-world conditions, the realism of the simulation environments plays a crucial role, as the system integration, verification, and validation are partly conducted through closed-loop simulations. In particular, the gap between simulation models and the real world must be taken into account when stimulating the system under test, since incorrect input data can lead to different behavior in the simulation compared to the real world. This paper provides a comprehensive overview of the challenges encountered in real-time simulation for highly automated driving using hardware-in-the-loop test systems. Starting from the general challenges of testing highly automated driving, we derive aspects that require special consideration in real-time simulations. Furthermore, we provide a detailed summary of the discrepancies between simulation and reality. Finally, we contextualize these discrepancies within the structure of hardware-in-the-loop and describe their influences and effects on the test system.