Bringing physics-based controller policies to virtual reality

The use of physics-based controllers in video games and virtual reality (VR) is now technically possible, but little attention has been paid to evaluating the movement quality generated by such controllers, particularly when high-quality motions are needed. Moreover, these controllers have been developed in research-oriented physics engines, which are not suitable for production projects, where game engines like Unity3D or Unreal Engine are preferred. Nowadays, it is easy to set up any element of a VR experience so users can interact with it physically, except for interactive VR characters. Can we extend this to animated VR characters? In this work, we studied whether physics-based character animation (PBCA) controllers can produce high-quality movement in VR-compatible environments (Unity3D, WebXR). Since VR experiences typically use rigged and textured characters, we evaluated the quality of the movement by capturing and manually retargeting eight walking styles to two rigged and textured characters, which served as reference animations for comparison with the motions generated by the PBCA controllers. We also found that showing the training results on rigged and textured characters helped highlight the controller policies’ animation inaccuracies. We evaluated the quality of the results, as well as the losses at each stage of the processing pipeline: data ingest, training and inference. For data ingest, we computed accuracy metrics related to the retargeting method used. For training, we computed accuracy metrics related to different training policies and different training environments. We developed new JAX implementations of state-of-the-art PBCA controllers (DeepMimic, AMP and ADD) which can be trained in MuJoCo-MJX in minutes, 50 times faster than in a traditional CPU-based training environment. The benefit of these particular implementations is that the resulting control policies can be readily transferred to Unity3D and support cross-platform compilation, including Android, something critical for use in VR production. For inference, we tested these controller policies in Unity3D and WebXR. Our results suggest that DeepMimic and ADD trained in MuJoCo-MJX generate controller policies with better animation fidelity than other methods. To simplify the comparison between controller policies, and different reference animations, we implemented an animation fidelity index (afi). Its results are analogous to other more common metrics, but afi is more intuitive to interpret and thus simplifies the comparison of training methods across the successive stages of the processing pipeline. We also did an expert comparison between the reference motions and the end result. Our results suggest that any use of PBCA in production projects should be approached with caution. Some inaccuracies readily visible to an animator were difficult to detect with the different metrics commonly used, and the overall quality of the results varied considerably depending on different choices made across the processing and delivery pipeline. Finally, we provide the animation data and code to facilitate the transfer of physics-based control research results to the artistic and production communities.