Synthèse des gestes co-verbaux : vers un modèle contrôlable et interprétable à l'aide d'une approche déterministe à base de graphe

Human communication is a multimodal process combining verbal and non-verbal dimensions, designed to foster mutual understanding. Gestures, in particular, enrich speech by clarifying meanings, expressing emotions and conveying abstract ideas. However, although advances in speech synthesis have made it possible to produce artificial voices close to human speech, existing systems often neglect visual cues, limiting the effectiveness and immersion of human-machine interactions. To fill this gap, this research developed STARGATE, an innovative model designed to integrate appropriate co-speech gestures into speech synthesis systems. The aim was to overcome the challenges of efficiency, interpretability and training on limited data, while producing gestures synchronized with speech. The STARGATE architecture is based on an autoregressive framework combining CNN networks for audio and text inputs, an ST-GCN network to encode the gesture history, and a biRNN decoder to generate gestures from these latent spaces. A key feature is the addition of an extra speaker embedding input, enabling gesture style customization according to the identity provided. The evaluation confirmed STARGATE's advantages over reference models such as StyleGestures and ZeroEGGS. Objectively, STARGATE achieved a significantly lower FGD, indicating better gesture quality. Subjectively, participants judged the gestures generated to be more humanlike and appropriate, particularly our 9-speaker model, which demonstrated speaker-dependent personalization capability. These results also showed that audio-only models are not sufficient to produce semantically relevant gestures, underlining the importance of integrating textual information. In addition, feedback from linguistic experts highlighted areas for improvement, notably gesture coherence and dynamics. This feedback guided the optimization of the model, reducing the FGD and improving its speed of inference, making it ideal for real-time use. Enhancements have strengthened STARGATE's capabilities, in particular the integration of finger movements, enabling the production of more expressive gestures, such as deictic gestures. The model was also generalized to several speakers, improving the quality of the gestures generated and enabling precise adaptation to each individual's style. This personalization was validated by a further evaluation, designed to analyze the ability of a multi-speaker model to produce different, yet consistent gesture identities. In-depth analysis revealed that STARGATE establishes a correspondence between textual embeddings and generated gestures. This proved the consistency of our model in its generation and understanding of semantic concepts when illustrated in metaphorical gestures. Our analyses of adjacency matrices have also led to the development of a gesture detector to identify gesture boundaries, with promising results. In conclusion, we think that STARGATE represents a major advance in co-verbal gesture synthesis, combining efficiency, control and interpretability into a deterministic model. It enhances human-machine interactions by enriching multimodal communication, while offering promising prospects for embodied conversational agents and assistive technologies.