Emotion-modulated interaction regression predicts static facial blendshape scores

Emotion-conditioned virtual avatars require a controllable bridge from affective cues to facial control parameters. This paper addresses static expression-parameter prediction: given a facial expression image and an emotion-class descriptor, the model estimates a 52-dimensional MediaPipe/ARKit-style blendshape-like vector. The 52-dimensional coefficients are automatic Face Landmarker estimates of ARKit-named scores from 2D images. Because training and evaluation share this extractor, the reported metrics measure agreement with those automatic scores. Predicted vectors remain candidate control profiles for a subsequent documented retargeting step. We propose EMIRA (Emotion-Modulated Interaction Regressor for Avatars), a lightweight class-conditioned interaction regressor that combines a 272-dimensional handcrafted image descriptor, emotion-class prototypes, and explicit class-conditioned interaction features. At inference time, we additionally evaluate a feature-shift fallback strategy that switches to an emotion prior when the feature-shift score exceeds a training-set quantile threshold, and we analyze its output-selection behavior under feature-shift diagnosis. The Yale Face Database is used as a small four-class feature-shift diagnostic. On an internal static-expression benchmark with 13,357 valid samples, EMIRA reduces RMSE by 14.07% over a rule baseline and by a modest 4.44% over image-only regression. Paired bootstrap comparisons over 2003 test samples show positive deltas against rule, image-only, early-fusion, and same-dimensional capacity-control baselines. A class-balanced diagnostic over five reliable emotion classes keeps a 4.31% RMSE gain over image-only regression. A per-blendshape mechanism diagnostic further shows that 83.90% of the positive per-dimension error reduction over image-only concentrates in mouth controls. These results support class-conditioned interaction on a given visual branch as a useful representation for static MediaPipe blendshape-score prediction rather than a broadly applicable fusion architecture, with the reported RMSE reductions measuring agreement with automatic MediaPipe scores rather than independently annotated animation quality.

Authors

Institutions

Publication Details

Journal
Scientific Reports
Published
2026-11-03
DOI
https://doi.org/10.1038/s41598-026-78607-x
Primary Topic
Face recognition and analysis
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Emotion-modulated interaction regression predicts static facial blendshape scores

Jin Zhang
Scientific Reports
Face recognition and analysis
article

Emotion-modulated interaction regression predicts static facial blendshape scores

Jin Zhang
article en

Abstract

Emotion-conditioned virtual avatars require a controllable bridge from affective cues to facial control parameters. This paper addresses static expression-parameter prediction: given a facial expression image and an emotion-class descriptor, the model estimates a 52-dimensional MediaPipe/ARKit-style blendshape-like vector. The 52-dimensional coefficients are automatic Face Landmarker estimates of ARKit-named scores from 2D images. Because training and evaluation share this extractor, the reported metrics measure agreement with those automatic scores. Predicted vectors remain candidate control profiles for a subsequent documented retargeting step. We propose EMIRA (Emotion-Modulated Interaction Regressor for Avatars), a lightweight class-conditioned interaction regressor that combines a 272-dimensional handcrafted image descriptor, emotion-class prototypes, and explicit class-conditioned interaction features. At inference time, we additionally evaluate a feature-shift fallback strategy that switches to an emotion prior when the feature-shift score exceeds a training-set quantile threshold, and we analyze its output-selection behavior under feature-shift diagnosis. The Yale Face Database is used as a small four-class feature-shift diagnostic. On an internal static-expression benchmark with 13,357 valid samples, EMIRA reduces RMSE by 14.07% over a rule baseline and by a modest 4.44% over image-only regression. Paired bootstrap comparisons over 2003 test samples show positive deltas against rule, image-only, early-fusion, and same-dimensional capacity-control baselines. A class-balanced diagnostic over five reliable emotion classes keeps a 4.31% RMSE gain over image-only regression. A per-blendshape mechanism diagnostic further shows that 83.90% of the positive per-dimension error reduction over image-only concentrates in mouth controls. These results support class-conditioned interaction on a given visual branch as a useful representation for static MediaPipe blendshape-score prediction rather than a broadly applicable fusion architecture, with the reported RMSE reductions measuring agreement with automatic MediaPipe scores rather than independently annotated animation quality.

Scientific Reports
Wuhan University of Technology (CN)
Openalex Percentile: Top 19%
Face recognition and analysis
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon
Emotion-modulated interaction regression predicts static facial blendshape scores — Jin Zhang · Scientific Reports (2026) | TGRS Research Map | TGRS