Modified Convolutional Neural Network with Multiple Features for Multimodal Sarcasm Detection

Due to the increasing prevalence of social media and online interactions, there's a growing need for analytical models that can interpret the complex and diverse forms of communication often found on these platforms, particularly in identifying sarcasm. Contemporary studies utilize multi-stage models and advanced techniques for extracting semantic information, often relying on single-mode encoders. However, these models frequently encounter difficulties in efficiently fusing and aligning multi-modal representations. This paper proposes a novel multimodal sarcasm detection using the MCNN model. This work utilizes three modalities like text, image and audio. The input text undergoes preprocessing through tokenization and stemming. The input image undergoes preprocessing through Gaussian filtering, while the input audio is pre-processed via median filtering. Subsequently, features such as TF-IDF from text, Modified LGIP from images, and MFCC-based features from audio are extracted. Following the feature extraction, the detection is done by training these features in the proposed Modified Convolutional Neural Network (MCNN). The MCNN model effectively detects the sarcasm.The accuracy ratings of the suggested model ranged from 0.842 to 0.936.