Multimodal Data Generation for Audio and Video Interaction

Multimodal Large Language Models (MM-LLMs) have made significant progress in the field of artificial intelligence, which not only inherit the advantages of traditional Large Language Models (LLMs) in text processing, but also realize the ability of cross-modal understanding and generation. This paper, based on the existing framework of MM-LLMs, realizes a cross-modal understanding and generation of LLM on audio and video. The model has four layers: modal encoder layer, input projection layer, LLM layer and modal generator layer. This model improves the input projection layer of the general MM-LLM architecture and the alignment between the LLM and the modal generator, and fine-tuning the modal generator to make the architecture more suitable for the task requirements of the model.