Orchestrating Generative Models for Synthesizing Domain-Specific Vision–Language Instructions

Vision-Language Models (VLMs) have demonstrated remarkable performance in general multimodal tasks, yet their effectiveness significantly deteriorates when applied to specialized domains such as electric power systems due to the scarcity of domain-specific data and expert annotations. To bridge this gap, we propose a novel, scalable pipeline for generating a high-fidelity and diverse domain-specific dataset. Our method comprises two sequential phases: (1) Feeding the collected seed images together with domain-specific edit prompts to an advanced generative image editing model, which produces diverse synthetic images that preserve the identity of critical objects even under complex editing instructions; and (2) Filtering out low-quality synthetic images by prompting a VLM with quality criteria, and subsequently using the VLM to construct domain-aware captions and visual instructions for the remaining synthetic images. The resulting synthetic dataset is used to finetune mainstream VLMs (e.g., Qwen-VL and InternVL) via an instruction-tuning paradigm. Extensive experiments demonstrate that the VLMs finetuned on our synthetic instructions significantly outperform those finetuned on the original visual instructions. Notably, models finetuned on our editing-augmented visual instructions achieve up to a 25% increase in F1-score on object grounding tasks relative to models trained solely on seed data. Furthermore, our synthesized data achieves up to a 13.66% relative gain on the VQA task compared with the original seed data. Our study thus provides an effective methodology for synthesizing domain-specific visual instructions, enabling VLMs to adapt to specialized industrial domains.