Multi-modal fusion for billboard categorization in video frames: A text and image-based approach

Abstract Traditional multimedia classification techniques rely on analyzing either its features or the associated annotated textual information. In this paper, we introduce a technique that leverages deep learning methodologies to extract both low-level visual features and high-level semantic information from the billboard images. We propose a multi-modal hybrid fusion model that integrates text and image features to categorize billboards in video frames into food, sports, and miscellaneous categories. Achieving a 5% accuracy improvement over image-based models and 10% over text-based models, our model demonstrates robust generalization across diverse datasets, benefiting advertising, media, and content creation industries.