Multimodal Artwork Topic Modeling via Fine-Tuned Clip and Knowledge-Driven Prompts

We propose a novel multimodal topic modeling framework to extract and explain latent themes in extensive collections of digitized artworks. Our approach leverages CLIP's contrastive pre-training to encode images and textual metadata into a shared semantic space. We fine-tune CLIP on a domain-specific dataset built from <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathcal{A}rt \mathcal{G}raph$</tex>, an art-domain knowledge graph containing over 100k artworks enriched with curated metadata. Using the resulting multimodal embeddings, we perform clustering to uncover latent visual topics and associate each cluster with descriptive terms via cosine similarity with templated textual prompts. Finally, to further interpret the discovered topics, we employ LLaVA to generate textual summaries based on representative images. Our framework demonstrates promising performance in terms of topic coherence and diversity, evaluated through both visual and textual metrics. The method is unsupervised, easily adaptable, and provides interpretable outputs, making it suitable for applications in digital humanities, cultural heritage analysis, and content-based art retrieval.