Multi-Model Workflows for Advanced Visual Understanding

This paper introduces a novel task-agnostic system for visual question answering, designed to address the limitations of existing visual-language models in scenarios requiring fine-grained reasoning and contextual understanding. By integrating computer vision, natural language processing, and retrieval-augmented generation, the system dynamically selects models and interprets diverse visual inputs without necessitating task-specific retraining. It leverages scene graph extraction and a model multiplexer to process visual content contextually, while an instruction execution engine transforms multimodal data into actionable insights. The system emphasizes explainability through annotated outputs and textual responses, enhancing user trust and comprehension. Its modular design ensures scalability and adaptability, enabling seamless incorporation of new models and datasets for evolving applications. This work represents a significant step forward in visual question answering, offering a transparent and versatile solution tailored to real-world complexities. The complete source code can be found at https://github.com/NVIDIA/Multi-Model-Workflows.