Development of multimodal English translation system based on biomechanics

With the increasing demand for translation quality, improving the translation quality of English translation systems has become a severe challenge. Traditional English translation systems mostly use single-modal information of text for translation, ignore multimodal information such as speech, and fail to combine the biomechanical movement data of the mouth, throat, and tongue, resulting in poor English translation quality. This paper develops an English translation system based on text, speech and biomechanical motion data, using the translation modeling of Transformer and the temporal feature processing advantages of BiLSTM (Bidirectional Long Short-Term Memory). The study first used MFCC to extract speech features and the spectral features of the audio, and used oral pressure sensors, optical sensors and other sensors to synchronously collect the students’ oral opening angles and lip curvature, vocal cord vibration frequency, glottal opening and closing degree, tongue tip position, tongue back curvature and other biomechanical motion feature data, and then input the features into the BiLSTM model for time series modeling and the time series relationship between speech and biomechanical features. Then, a multimodal Transformers framework was constructed, and text, speech, and biomechanical data were integrated using a weighted fusion method, and the multi-head attention mechanism was combined for translation. Finally, the BiLSTM model and the multimodal Transformers framework were integrated in series, and the module was designed and integrated into the English translation system. The experiment was based on the LibriSpeech ASR corpus. The results showed that the multimodal combination of text + audio + biomechanical information performed best, reaching 0.82, 0.76 and 0.79 in BLEU, METEOR, and ROUGE-L indicators respectively, which were 0.06 higher than the multimodal combination without the introduction of biomechanical information, and the loss of emotion transfer was only 0.01. The experimental results show that combining biomechanical information can significantly improve the translation quality of the multimodal English translation system and maintain the original emotion of the language.