Generalized domain prompt learning for accessible scientific vision-language models

<h2>Abstract</h2> Large-scale vision-language models have shown remarkable success in general vision tasks, inspiring the development of domain-specific models. However, creating these models requires substantial investments in annotated data, computational resources, and energy, making such endeavors largely accessible to industrial giants. This resource-intensive nature of model development inadvertently stifles academic research, particularly for smaller research groups, and limits the diversity and growth of the field. To address these challenges and promote more sustainable and equitable research, we introduce the generalized domain prompt learning framework. This framework enables the efficient transfer of robust recognition capabilities from natural vision to specialized domains without the need for large datasets or heavy computational overhead. By leveraging small-scale domain-specific foundation models and minimal prompt samples, the framework enriches the language component with domain-specific knowledge through quaternion networks, revealing cross-modal relationships between specialized vision features and natural vision-based contextual embeddings. Simultaneously, it drives the vision component toward domain-specific tasks via hierarchical propagation of vision prompts, grounded in well-matched vision-language relationships. To maximize the domain adaptation potential of these models, we also propose a novel low-rank adaptation technique. Extensive experiments across diverse domains—including remote sensing, medical imaging, geology, synthetic aperture radar, and fluid dynamics—demonstrate the effectiveness of the framework, achieving state-of-the-art domain recognition performance within a prompt learning structure. Our work presents a pathway for inclusive and sustainable research that bridges the gap between academia and industry, empowering smaller research groups and promoting broader access to cutting-edge advancements in the context of future sustainable development.