Strengths-Leverage Chain-of-Thought: Enhancing Multimodal Reasoning with LLM and LMM

AI systems have long sought to replicate humans’ complex multimodal reasoning capabilities. Recent advancements in large language models (LLMs) showcase significant progress, particularly in multi-step reasoning within language. These models utilize a technique known as the Chain-of-Thought (CoT), which effectively simulates human cognitive processes, to achieve this enhanced reasoning ability. However, transferring these advances to multimodal contexts poses more significant challenges. Recent studies indicate that despite their sophistication, the most advanced LMMs still struggle with compositional visual reasoning, particularly in understanding object attributes and relationships. Furthermore, hallucinations in models are exacerbated by the interaction of multimodal inputs. The strength of the LMM lies in processing visual tasks, including extracting scene graphs, while the LLM excels in handling textual information and leveraging internal knowledge to answer questions. Inspired by these and to address the above issues, we propose Strengths-Leverage Chain-of-Thought (SLCoT) prompting, a novel zero-shot Chain-of-Thought prompting method. SLCoT deconstructs questions while extracting scene graphs (SG) from images via the LMM. Finally, the LLM integrates the SG and sub-answers to derive a comprehensive rationale. By leveraging the strengths of both models in cross-modal tasks, we enhance AI’s multimodal reasoning capabilities. Extensive experiments demonstrate the efficacy of our proposed framework, showing significant improvements in multimodal CoT performance across complex visual reasoning datasets without the need for fine-tuning.