Structured Intention Generation with Multimodal Graph Transformers: The MMIntent-LLM Framework

In the task of answering questions related to electricity knowledge, accurately understanding user intentions is fundamental to building an effective reasoning process. Questions in this domain often cover complex issues such as troubleshooting, electricity bill consultation, and electricity safety. User queries frequently involve multiple aspects and varying levels of information. Traditional single-modal methods struggle to fully address these diverse needs. Intention understanding plays a crucial role in constructing a clear and logical reasoning chain by accurately identifying the user’s core requirements. Through in-depth analysis of user intentions, the system can better organize and reason with multimodal information such as text, voice, and images—leading to logically coherent and accurate responses. Intention understanding not only enhances the accuracy of the reasoning process but also significantly improves the response efficiency and service quality of the electricity knowledge question-and-answer system. Current approaches to intention analysis primarily rely on classification-based methods, which limit the flexibility and richness of intent understanding. To broaden the scope and depth of user intent recognition, we introduce MMIntent-LLM, a novel framework that combines T5 with a Graph Transformer to generate structured intent representations from multimodal social media content. Our approach introduces three key innovations: (1) a structured intention reasoning framework based on ATOMIC, which provides a systematic method for decomposing intent generation into interpretable components; (2) a graph-based alignment mechanism for multimodal data (text and images) that ensures semantic consistency across modalities; and (3) an adaptive fine-tuning strategy that effectively transfers knowledge from pre-trained language models to the specific intent generation task. Extensive experiments on our multimodal social intention dataset show that MMIntent-LLM achieves state-of-the-art performance, improving the average BERT score by 8.7% and human evaluation scores by 12.3% compared to baseline methods.