From Faces to Politics: Vision-Language Models (Sometimes) Link Visual Demographic Characteristics to Ideological Labels

Abstract When foundation models analyze political content, do they use demographic characteristics as shortcuts for ideological attribution? We conducted detailed experiments with GPT-4o-mini and validated key findings across GPT-4o and LLaVA , using identical, ideologically neutral campaign advertisements with systematically varied candidate demographics. All models consistently attributed more liberal ideologies to women than men. These effects exceeded real-world gender differences from a nationally representative survey. However, racial associations differed by model: strong in GPT-4o-mini (where Black candidates received substantially more liberal attributions), attenuated in GPT-4o , and insignificant in LLaVA . These demographic effects persisted across temperature settings, prompt variations, and even explicit debiasing instructions in GPT-4o-mini . Our findings reveal that visual demographic features can shape AI outputs in ways that vary across models, with implications for applications such as content classification.