Applications of Large Language Models in Multimodal Learning

In this paper, we provide a systematic review of the emerging field on applications for Large Language Models (LLMs) in multimodal learning, especially how such methodologies help improve orchestrated task performance by integrating different modalities like images, text, and audio. Multimodal learning is a field where we combine various types of data to make models learn multiple attributes and generate meaningful outputs. It is widely applied in image captioning, cross-modal retrieval, sentiment analysis, and speech recognition. It reviews the main multimodal learning approaches, such as feature extraction, modality alignment, and fusion strategies (early fusion, late fusion, and hybridization), and the performance of LLMs in cross-modal tasks. It highlights the present technological challenges, emphasizing concerns regarding computational resource utilization, model complexity, as well as a lack of multimodal fusion. Lastly, the article provides some suggestions for future applications on how to better integrate modalities and few-shot learning in cross-modal generation models. It also discusses ways to make multimodal machine translation systems run faster using less distributed computational power.