Real-Time MRI Synthesis from Text Using a Two-Stage Pipeline for Pronunciation Training

Providing visual or animated information on how to position and move the tongue and other speech articulators is beneficial for children with hearing impairments to improve their pronunciation. However, since these articulatory movements occur inside the mouth, they are not visible. Real-time MRI of the midsagittal plane of the mouth can be used to extract this information. However, acquiring MRI data for every word is both time-consuming and impractical. This work proposes a method to synthesize MRI sequences for a given text. The approach involves a two-stage pipeline. In the first stage, text-to-speech synthesis is performed to replicate the speaker’s voice characteristics. In the second stage, the synthesized speech is mapped to an MRI sequence using a speech-MRI mapping model trained on the speaker recordings. The synthesized speech and corresponding rtMRI sequences are then integrated to generate a synchronized rtMRI video, providing a simultaneous representation of both articulatory movements and acoustic output. A key advantage of the proposed method is that it eliminates the requirement for phoneme-to-MRI frame alignment during the training of phoneme-to-MRI mapping models, a process that is frequently error-prone when relying on forced-alignment techniques. The Tortoise-TTS system is employed for speech synthesis from the text, ensuring that the generated speech matches the target speaker’s voice characteristics. A bidirectional Long Short-Term Memory network is trained to map speech to the corresponding MRI image sequence. To evaluate the synthesized MRI sequences, both objective measures, such as Frechet Inception Distance, and subjective evaluations are utilized. The mean FID scores ranged from 94.23 to 135.28, comparable to those obtained using other forced alignment-based methods. Subjective evaluations on a 1-to-5 scale yielded an average score of 3.5. The results demonstrate that the generated MRI sequences are realistic and closely resemble natural recordings, with performance metrics indicating a satisfactory level of accuracy and quality.