Exploring The Potential of Vision-Language Models for Pure-Image and Text-Guided-Image Saliency Prediction

We introduce VLSal, a saliency prediction framework that leverages Vision-Language Models (VLMs) to unify pure-image and text-guided-image saliency prediction tasks and achieve high performance in both. We extract visual features from the visual encoder and retrieve the corresponding visual token features from the language decoder, which serves as a natural feature fusion mechanism. These features are then processed through a U-Net-based saliency decoder to generate accurate saliency maps. To efficiently adapt the large-scale pretrained model, we apply Low-Rank Adaptation (LoRA) finetuning, reducing computational costs while preserving performance. Extensive experiments on benchmark datasets, including SALICON, MIT1003, and TIS, demonstrate that VLSal outperforms existing methods in both pure-image and text-guided-image saliency prediction.