Birds, Bias, and Better AI: Improving Vision-Language Attention with Probabilistic Adapters : A post hoc probabilistic approach to attention guided learning from frozen vision language models

Vision-language models (VLMs) have demonstrated powerful capabilities across a wide range of tasks, but their deterministic nature limits their ability to express uncertainty, a key factor in robustness and fairness. This project explores the integration of two post-hoc probabilistic uncertainty estimation techniques, ProbVLM and BayesVLM, into the GALS (Guiding visual Attention with Language Specification) framework. ProbVLM leverages a probabilistic adapter trained on top of CLIP with a ResNet backbone, producing heteroscedastic embedding distributions and using Grad-CAM to generate attention maps. BayesVLM, on the other hand, applies a Laplace approximation to the projection layers of a ViT-based CLIP model, analytically propagating uncertainty to yield probabilistic cosine similarity scores, with RISE used for attention visualization. The experiments, conducted on the Waterbirds dataset under the GALS protocol, demonstrate that both probabilistic methods improve performance compared to the original GALS implementation. This improvement is evident in classification accuracy and in fairness-related metrics, indicating that uncertainty-aware attention mechanisms can better mitigate spurious correlations. These results highlight the promise of post-hoc probabilistic VLMs in enhancing both the interpretability and reliability of vision-language models in fairness-critical tasks.