Efficient Fine-tuning of Vision Language Models and Applications

Pre-trained vision-language models are now widely adopted in multimodal AI, demonstrating strong performance across a range of tasks including visual question answering, image captioning, and cross-modal retrieval. However, adapting these models to downstream applications remains computationally intensive and resource-demanding. In this talk, I present a novel framework, Double-Filter, designed to streamline the fine-tuning process of VLP models by reducing redundancies in both feature inputs and network architecture. The approach introduces two key innovations: (i) we introduce a new method for selecting image patches. This involves separating the background from the foreground to filter out less relevant areas, followed by a refined process that focuses on identifying the most meaningful parts of the image, and (ii) a Fine-Grained Architecture Layer Filter, which employs an adjustable genetic algorithm to identify and prune redundant sublayers, rather than entire blocks, thereby preserving performance while improving efficiency. I will detail this approach and assess its influence on the effectiveness of question answering. Looking ahead, I will outline future directions, including decoupled fine-tuning strategies, which aim to separate modality-specific adaptation from cross-modal alignment, and the integration of personality-aware question answering, where user traits and interaction styles inform the model response generation. These developments hold promise for building more efficient, personalised, and scalable multimodal systems across domains such as education, healthcare, and interactive AI.