Analyzing and Categorization Developer Intent on Twitch Live Chat

Modern knowledge sharing platforms, such as Twitch, have gained significant popularity among developers. Twitch allows streamers (programmers) to broadcast real-time programming tasks across various domains. Viewers engage with streamers through live-chat messages, posting comments to other viewers or asking questions. However, the majority of comments are unhelpful, including spam and unrelated discussions. Streamers are faced with the challenge of identifying useful comments, either by reading the entire chat log, which diverts their attention from coding, or by periodically skimming through, risking the chance of missing important comments. Objective: Therefore, it is crucial to investigate and understand the content of viewer comments in live-streamed programming on Twitch. Method: In this paper, we present a manually labeled dataset of Twitch live-chat messages from programming streams. We conduct experiments to evaluate the accuracy of transformer-based models in automatically classifying these comments. Results: Our results illustrate that fine-tuning through training the BERT-base model with a specific preprocessing technique, @mention removal, can effectively classify Twitch live-chat messages with an F-score of 0.84. Conclusions: The outcomes of this work provide valuable insights for future research leveraging our labeled dataset, and the trained model can serve as a foundational approach for analyzing live stream platforms like Twitch.