Grounding Image Understanding to Oil and Gas Product Manuals: Refining LLaVA through Contextual Instruction Tuning

The significance of multimodal models lies in their enhanced capabilities to parse, understand, and reason on complex documents that contain a mixture of text, images, tables, and other components. These models have shown remarkable proficiency in providing coherent and contextually accurate descriptions and analyses, thus significantly advancing document processing tasks. However, to maximize their utility in specific industries, it is crucial to adapt these multimodal models to domain-specific tasks, where specialized knowledge and data are required for precise interpretation and application. In this paper, our approach to multimodal document processing bridges the gap between proprietary large-scale models and smaller open-source models. By combining the robust embedding capabilities of proprietary models with the modular and resource-efficient architecture of open-source models, the proposed method aims to enhance domain-specific image description tasks. Our approach focuses on fine tuning multimodal models on domain-specific data, particularly in the oil and gas industry, by generating data grounded in domain knowledge through the use of proprietary multimodal models. By adapting and fine tuning open-source smaller-scale models on our proprietary datasets, we were able to demonstrate the potential for significant advancements in generating precise image descriptions and answering detailed questions related to image content, thus providing meaningful insights and facilitating downstream applications.