See Spot Run: Real-Time Monitoring and Control of Boston Dynamics Spot from Virtual Reality

The integration of Virtual Reality (VR) Environments and robotics platforms into digital twins offers unique ways of engaging with physical and digital spaces. However, application of Virtual Reality Environments is often limited by the challenges associated with cybersickness, a type of motion sickness that occurs when the brain receives conflicting motion and vision signals. Thus, bridging the gap between these two technologies requires the development of a robust platform for real-time interaction and monitoring. This work presents a novel approach to creating a real-time digital twin of the Spot robotic platform within a VR environment using the Unity game engine. To avoid cybersickness, VR environments typically require frame rates of 30 Frames per Second (FPS) or higher, but this is challenging due to variable MQTT message rates, which can lead to inconsistencies in the robot's representation within VR. However, the VR Environment itself can be leveraged to extrapolate the missing message values. To assess the accuracy of the extrapolated values from the VR environment, we modeled the robot's joint position and velocity data using Cubic Splines finding overall the VR environment excelled at modeling joint Positions (<tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$R^{2}=.980 \pm. 035$</tex>) but struggled modeling joint velocities (<tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$R^{2}=.873 \pm. 131$</tex>). Our results demonstrate the feasibility of real-time VR-based monitoring and control of the Spot platform, with promising implications for remote teleoperation, simulation, and other research applications.