Avatar Emotion: Real-Time Emotion Detection and Expression for Sign Language Interpreting Avatars

The integration of emotion detection within avatar communication systems has overhauled delivery of emotionally enhanced content, particularly for applications in sign language interpretation. Yet fundamental difficulties persist regarding effective detection of emotion from free-form speech or text and mapping these dynamically into detailed expressions from animated avatars. Within this paper, we introduce a framework based on deep learning that relies on a transformer-based emotion identification model combined with a keypoint projection mechanism in order to connect the two ends. The first step of the system is employing fine-tuned DistilBERT embeddings to recognize emotions from both spoken or transcribed input. These emotion identifiers (e.g., happy, angry, sad) are submitted to a pose decoder that shares the same alignment as avatar facial mesh structures for synchronized emotional expression and gestures. The architecture is also coupled with a progressive transformer to provide efficient and contextually aware generation of avatar animations. The method facilitates real-time emotional fidelity in expressions by the avatar, improving accessibility and engagement for applications such as virtual interpreters, digital assistants, and sign language generation systems. Experimental evaluation proves that the method outperforms current baselines in terms of emotion accuracy and visual expressiveness and can thus be considered an effective approach for emotion-to-avatar mapping.