Improved Speech Synchronized Talking Face Generation from an Image and an Emotion Condition

By granting computers the ability to communicate emotions through facial expressions, human-computer interactions can be made more organic and natural. This study analyses and refines an emotional talking face model that integrates face images, categorical emotions, and speech signals to produce synchronized facial expressions. Incorporating a squeeze-and-excitation network, introducing non-linearity, and enhancing effectiveness led to a significant improvement in the model's performance. A Blind Image Super-Resolution Generative Adversarial Network enhanced video resolution, while multi-task cascaded convolutional neural networks accurately detected and utilized the eye region. Evaluation of the model employed Fréchet Inception Distance score for objective assessment and Mean Opinion Score for subjective evaluation. Results conclusively demonstrate substantial enhancements in accuracy and performance facilitated by these network additions. This work was completed as a B.Tech final year project at the Department of Electronics and Communication Engineering, National Institute of Technology Calicut (2022-2023), under the supervision of Dr. Ameer P.M.