Investigating F0 Estimation in Speech Synthesis from Real-time MRI Articulatory Data

This study investigates the feasibility of estimating the fundamental frequency (F<inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">0</inf>) directly from individual MRI images, inspired by prior successes in speech synthesis using real-time MRI (rtMRI) videos. Building on our previous work, we developed a model for F<inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">0</inf> estimation from single midsagittal MRI frames, exploring whether F<inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">0</inf> could be derived from static vocal tract configuration. Using a dataset of two male and two female Japanese speakers, we determined that accurate F<inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">0</inf> estimation from single MRI images is achievable, with results comparable to those obtained using speech synthesis methods. Optical flow analysis revealed subtle articulatory movements related to F<inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">0</inf> control, including the upward and forward movement of the larynx and tongue with increasing F<inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">0</inf> and a forward-rotating motion around the hyoid bone area in high F<inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">0</inf> regions. Notably, distinct patterns were observed in male speakers at low F<inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">0</inf> ranges, suggesting individual variations in F<inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">0</inf> control.