NePHIM: A Neural Physics‐Based Head‐Hand Interaction Model

Abstract Due to the increasing use of virtual avatars, the animation of head‐hand interactions has recently gained attention. To this end, we present a novel volumetric and physics‐based interaction simulation. In contrast to previous work, our simulation incorporates temporal effects such as collision paths, respects anatomical constraints, and can detect and simulate skin pulling. As a result, we can achieve more natural‐looking interaction animations and take a step towards greater realism. However, like most complex and computationally expensive simulations, ours is not real‐time capable even on high‐end machines. Therefore, we train small and efficient neural networks as accurate approximations that achieve about 200 FPS on consumer GPUs, about 50 FPS on CPUs, and are learned in less than four hours for one person. In general, our focus is not to generalize the approximation networks to low‐resolution head models but to adapt them to more detailed personalized avatars. Nevertheless, we show that these networks can learn to approximate our head‐hand interaction model for multiple identities while maintaining computational efficiency. Since the quality of the simulations can only be judged subjectively, we conducted a comprehensive user study which confirms the improved realism of our approach. In addition, we provide extensive visual results and inspect the neural approximations quantitatively. All data used in this work has been recorded with a multi–view camera rig. Code and data are available at https://gitlab.cs.hs‐rm.de/cvmr_releases/HeadHand .