Beyond Words: Multi-modal Chat Summarization using Text with Emoji and GIF Emotion Analysis

This paper presents a novel approach that signifies a methodological change in the way that a conversation is summarized in a chat. The term “Beyond Words” implies that the chat summarization process transcends conventional text-based techniques. It suggests that in addition to the textual content, the summarization takes into account extra elements like the emotions conveyed by emojis and GIFs. “Beyond Words” highlights a more thorough and inclusive method of summarizing chats by considering visual cues like emojis and GIFs that enhance communication in addition to written text. Our methodology aims to provide a more comprehensive and expressive representation of conversations by integrating these multimodal components. The paper describes the development and fine-tuning of transformer model using available familiar datasets. Furthermore, incorporating emotion detection from GIFs using a fine-tuned CNN pre-trained model, as well as emotion analysis of chat emojis, improve the summarization process. The results show that our approach is effective at capturing the essence of conversations, including not only words but also the emotive and visual dimensions inherent in modern digital communication. This project presents a fresh approach to chat summarization, allowing users to understand lengthy conversations in a clear and informative format.