Research on generation algorithm of multimodal dialogue system integrating speech, vision and text

This paper studies a multimodal dialog system generation algorithm that integrates speech, vision and text information to improve the naturalness and accuracy of human-computer interaction. For audio data, MFCC frequency cepstrum coefficient (MFCC) technology is used to extract speech features, and the semantic changes in time series are captured by stacking LSTM networks to realize the understanding of speech content. For visual information, FaceNet model is used to process facial images, extract high-dimensional face feature vectors, and dynamically adjust the importance of visual input combined with attention mechanism to enhance the perception ability of dialogue system. The text part is semantically encoded using a pre-trained BERT model to ensure efficient parsing of complex language structures. On this basis, a multi-modal fusion layer is designed, and Transformer architecture is used to integrate the information of the three modes to ensure the consistency and coherence of the dialogue process. The experimental results show that the proposed algorithm significantly improves the response quality of the dialogue system and provides strong support for building an intelligent interactive platform.