When Less is More: A Sparse Facial Motion Structure for Listening Motion Learning

Effective human behavior modeling is critical for successful human–robot interaction. Current state-of-the-art approaches for predicting listening head behavior during dyadic conversations employ continuous-to-discrete representations, where continuous facial motion sequence is converted into discrete latent tokens. However, nonverbal facial motion presents unique challenges owing to its temporal variance and multimodal nature. State-of-the-art discrete motion token representation struggles to capture underlying nonverbal facial patterns making training the listening head inefficient with low-fidelity generated motion. This study proposes a novel method for representing and predicting nonverbal facial motion by encoding long sequences into a sparse sequence of keyframes and transition frames. By identifying crucial motion steps and interpolating intermediate frames, our method preserves the temporal structure of motion while enhancing instance-wise diversity during the learning process. Additionally, we apply this novel sparse representation to the task of listening head prediction, demonstrating its contribution to improving the explanation of facial motion patterns.