MEKA: Multimodal External Knowledge Augmentation for Social Media Image Captioning

Multimodal Large Language Models (MLLMs) have shown promising results in image captioning tasks. However, their effectiveness for social media content is limited by static training data and inability to capture real-time context, especially for events that postdate their training. We present MEKA (Multimodal External Knowledge Augmentation), a novel framework that enhances MLLM-based image captioning for social media posts by dynamically integrating external knowledge sources. MEKA combines web search, reverse image search, and entity recognition to provide rich contextual information without relying on the internal knowledge of the MLLMs. Our experimental results demonstrate that MEKA significantly outperforms state-of-the-art models such as GPT-4-turbo, achieving a 97% win rate in human evaluations. Through ablation studies, we show that while entity recognition significantly contributes to caption quality, the impact of query generation is minimal in the current implementation. MEKA's scalable architecture and ability to incorporate real-time information make it particularly effective for social media content where understanding cultural context, humor, and current events are crucial.