A Novel Approach for Hashtag Topic Classification with Multimodal Features and VLMs

The rapid expansion of Instagram content marked by a large volume of photos and comments demands effective methods for discovering topics. While hashtags facilitate the grouping of posts, categorizing these hashtags into specific topics is challenging task due to the multimodal and noisy nature of content on the platform. In this paper, we introduce a novel approach that leverages multimodal features and Vision-Language Models (VLMs) for the topic classification of hashtags. To achieve this, we develop a dataset InstaHash which includes 464 hashtags and 12,345 posts spanning five topic categories: army, politics, sports, judiciary, and religion. We extract text features using the TF-IDF algorithm and visual features using HSV, SIFT, ResNet50, and VGG16 algorithms. Additionally, we introduce 34 binary features derived from the VLM of LLAVA through prompt engineering. These prompt engineering features examine the six aspects of images such as people, symbols and objects, buildings, events, uniforms, and embedded text. We train four classifiers of Support Vector Machine (SVM), Logistic Regression (LR), Decision Tree (DT), and Convolutional Neural Network (CNN) for topic classification of hashtags. Our analysis reveals that combining text and visual features yields an accuracy of 0.89 for topic classification with the CNN classifier. However, by incorporating additional prompt features into the model, we achieve the highest accuracy of 0.96. Moreover, we notice that prompt features related to aspects of people and symbols are the most important features for the topic classification of hashtags.