Analyzing and Detecting Toxicities in Developer Online Chatrooms: A Fine-Grained Taxonomy and Automated Detection Approach

Developer online chatrooms serve as vital connections among developers in Open Source Software (OSS) projects. However, the presence of impatient and unfriendly users can disrupt the harmony within these chatrooms, leading to negative consequences such as reduced activity and attrition. To address this, we aim to first understand toxicity in developer chatrooms and further develop automated detection techniques. We collect chat messages from three active Gitter chatrooms, and conduct an in-depth analysis at the discussion thread level to examine intent and sentiments. Using a card-sorting method, we create a fine-grained taxonomy with seven toxicity categories and manually annotate a dataset of 5,158 threads. This enables us to gain insights into the nature of toxicity and identify shortcomings in existing detection methods. We further propose an automated binary toxicity detection approach that integrates textual features, non-textual features, and negative sentiment features derived from a Large Language Model (LLM). Our experimental results demonstrate an average F1-Score of 0.546, representing a significant 57.8 % improvement over the best-performing baseline. We also validate the effectiveness of incorporating non-textual and negative sentiment features.