Sentiment Analysis of Multimodal Content: A Fusion of Visual and Textual Clues

Multimodal data analysis is crucial in understanding complex information across various domains in today's digital society. It integrates textual, visual, and audio data, offering deep insights into human behavior and interactions. Utilizing advanced machine learning techniques like deep learning and fusion methodologies, multimodal data analysis drives innovation in industries such as marketing, entertainment, healthcare, and finance. This work introduces a novel sentiment analysis approach, emphasizing multimedia content interpretation using the CMU-MOSI dataset with CNN and BERT models and by combining the inputs through early and late fusion techniques. Early fusion integrates features from transcripts and video frames after which a variety of machine learning techniques, including Support Vector Machine, Logistic Regression, and Naive Bayes are used for classification, whereas late fusion uses a weighted average formula to balance the predictions. This method enhances sentiment analysis accuracy and multimedia content interpretation, benefiting areas like digital media analysis, psychology, and marketing.