AssistGPT: Towards Multi-modal Agent for Human-Centric AI Assistant

Recent advancements in Multi-modal Large Language Models have significantly enhanced the capabilities of AI assistants, enabling users to perform question-answering based on the content of images or videos. However, addressing complex visual-based tasks, particularly those involving fine-grained human actions from lengthy videos, remains a formidable challenge. To tackle this issue, we introduce AssistGPT, an MLLM-based agent that takes images and videos and a query as input and outputs response in an interleaved video, image, and text format, providing users with comprehensive answers. The core idea of AssistGPT is to approach visual QA as a multi-step reasoning problem. AssistGPT decomposes queries into sequential steps, employing action recognition, and temporal and spatial grounding models to search for information within visual inputs. This approach enables us to create a versatile multi-modal assistant that can help users with various human-centric tasks, such as daily activities, learning sports skills, and healthcare.