Evaluating Large Vision-Language Models for Visual Framing Analysis in News Imagery: A Theory-Driven Benchmark

This study evaluates and compares the effectiveness of multiple large vision-language models (LVLMs) for automated visual framing analysis in the context of news imagery about social movements. Specifically, we evaluate LVLMs (Gemma3-27B, GPT-4.1, InternVL3-14B, InternVL3-38B, and Qwen2.5-VL-72B) against human-annotated ground truth data, using both baseline prompts and a range of Chain-of-Thought (CoT) prompting strategies with increasing complexity (i.e., from simple to detailed to expert). Model performance is assessed across visual framing categories: conflict, peace, and solidarity, using standard evaluation metrics including F1-score and Cohen’s kappa. Our findings show that (1) CoT prompting improves model alignment with human annotations across most framing categories, especially for complex social cues like solidarity; (2) expert-level CoT prompts show the highest agreement with human coders; and (3) model performance varies by the specific model in focus, with InternVL3-38B consistently outperforming others. This study provides a scalable and theory-driven framework for applying LVLMs to visual content analysis in social science research.