Emotion-modulated interaction regression predicts static facial blendshape scores
Emotion-conditioned virtual avatars require a controllable bridge from affective cues to facial control parameters. This paper addresses static expression-parameter prediction: given a facial expression image and an emotion-class descriptor, the model estimates a 52-dimensional MediaPipe/ARKit-style blendshape-like vector. The 52-dimensional coefficients are automatic Face Landmarker estimates of ARKit-named scores from 2D images. Because training and evaluation share this extractor, the reported metrics measure agreement with those automatic scores. Predicted vectors remain candidate control profiles for a subsequent documented retargeting step. We propose EMIRA (Emotion-Modulated Interaction Regressor for Avatars), a lightweight class-conditioned interaction regressor that combines a 272-dimensional handcrafted image descriptor, emotion-class prototypes, and explicit class-conditioned interaction features. At inference time, we additionally evaluate a feature-shift fallback strategy that switches to an emotion prior when the feature-shift score exceeds a training-set quantile threshold, and we analyze its output-selection behavior under feature-shift diagnosis. The Yale Face Database is used as a small four-class feature-shift diagnostic. On an internal static-expression benchmark with 13,357 valid samples, EMIRA reduces RMSE by 14.07% over a rule baseline and by a modest 4.44% over image-only regression. Paired bootstrap comparisons over 2003 test samples show positive deltas against rule, image-only, early-fusion, and same-dimensional capacity-control baselines. A class-balanced diagnostic over five reliable emotion classes keeps a 4.31% RMSE gain over image-only regression. A per-blendshape mechanism diagnostic further shows that 83.90% of the positive per-dimension error reduction over image-only concentrates in mouth controls. These results support class-conditioned interaction on a given visual branch as a useful representation for static MediaPipe blendshape-score prediction rather than a broadly applicable fusion architecture, with the reported RMSE reductions measuring agreement with automatic MediaPipe scores rather than independently annotated animation quality.
Authors
- Jin Zhang
Institutions
- Wuhan University of Technology (CN)
Publication Details
- Journal
- Scientific Reports
- Published
- 2026-11-03
- DOI
- https://doi.org/10.1038/s41598-026-78607-x
- Primary Topic
- Face recognition and analysis
- Type
- article
- Field-Weighted Citation Impact
- 0.00