An Image Is Worth K Topics: A Visual Structural Topic Model with Pretrained Image Embeddings

Abstract Political scientists are increasingly interested in analyzing visual content at scale. However, the existing computational toolbox is still in need of methods and models attuned to the specific challenges and goals of social and political inquiry. In this article, we introduce a visual structural topic model (vSTM) that combines continuous image embeddings with a structural topic model. This has important advantages compared to existing approaches. First, continuous embeddings allow the model to capture the semantic complexity of images, which is relevant to the study of politics. A consequence of using continuous embeddings is that we can directly use off-the-shelf pretrained deep learning models, offering state-of-the-art performance in image analysis tasks while saving time, cost and resources. Second, the structural topic model provides the ability to analyze how topics and covariates are related, while maintaining a nuanced representation of images as a mixture of multiple topics. In our experiments, we show that the vSTM is able to identify topics that are interpretable, coherent and relevant to the study of online political communication.