DEEP BELIEF NETWORK (DBN) IMPLEMENTATION FOR MULTIMODAL CLASSIFICATION OF SENTIMENT ANALYSIS

In sentiment analysis, the use of multimodal data, consisting of a combination of images and text, is becoming increasingly important for understanding digital context. However, the main challenge lies in effectively integrating these two types of data into a single learning model. Deep Belief Network (DBN), with its capability to learn hierarchical data representations, is utilized to explore optimal strategies for multimodal sentiment analysis. The dataset includes 34,034 images from the FERPlus dataset to train the model in classifying emotions based on facial expressions, as well as 999 text and image samples obtained through crawling X. Experiments were conducted by comparing the performance of DBN with 2, 3, and 4 hidden layers across different test data sizes (10%-50%). The results indicate that the 3-hidden-layer configuration achieved the best performance, with a highest accuracy of 76% at a 20% test data size. Additionally, testing different learning rates (10⁻⁴ to 10⁻⁷) produced consistent results, but the fastest computation time was achieved with a learning rate of 10⁻⁴. Based on these findings, DBN with a 3-hidden-layer configuration and a learning rate of 10⁻⁴ is considered a more efficient alternative for multimodal sentiment analysis based on text and images.