Gelina: Unified Speech and Gesture Synthesis Via Interleaved Token Prediction
Human communication is multimodal, with speech and gestures tightly coupled, yet most computational methods for generating speech and gestures synthesize them sequentially, weakening synchrony and prosody alignment. We introduce Gelina, a unified framework that jointly synthesizes speech and co-speech gestures from text using interleaved token sequences in a discrete autoregressive backbone, with modality-specific decoders. Gelina supports multi-speaker and multi-style cloning and enables gesture-only synthesis from speech inputs. Subjective and objective evaluations demonstrate competitive speech quality and improved gesture generation over unimodal baselines.
Authors
- Laure Soulier (ORCID: https://orcid.org/0000-0001-9827-7400)
- Nicolas Obin (ORCID: https://orcid.org/0000-0002-5236-5306)
- Catherine Pélachaud (ORCID: https://orcid.org/0000-0003-1008-0799)
- Jonas Beskow (ORCID: https://orcid.org/0000-0003-1399-6604)
- Gustav Eje Henter (ORCID: https://orcid.org/0000-0002-1643-1054)
- Théodor Lemerle
- Téo Guichoux
- Shivam Mehta (ORCID: https://orcid.org/0000-0002-1886-681X)
Institutions
- Institut de Recherche et Coordination Acoustique Musique (FR)
- Sorbonne Université (FR)
- Sciences et Technologies de la Musique et du Son (FR)
- Université Paris 1 Panthéon-Sorbonne (FR)
- KTH Royal Institute of Technology (SE)
Publication Details
- Published
- 2026-04-21
- DOI
- https://doi.org/10.1109/icassp55912.2026.11464562
- Primary Topic
- Hand Gesture Recognition Systems
- Type
- article
- Field-Weighted Citation Impact
- 0.00
Funders
- Technology Development