Sentiment Analysis of #Meanspo Tweets: Humans vs. Automatic Classifications

With the increasing adoption of automatic text classifications driven by AI, there is a growing need to explore their safe and accurate applications, particularly in sensitive online communities. Sentiment analysis of X (formerly known as Twitter) data is widely used by researchers to automatically categorize textual data, providing valuable insights into the content of specific online communities. In this study, we investigate the effectiveness of automatic sentiment classification models (TextBlob and Vader) by analyzing n=6930 #meanspo tagged tweets from 2020 to 2022 from X. This community is known for promoting harmful eating disorder related content, often in a harsh and derogatory manner. By comparing these models with human coding, our analysis reveals significant limitations in the models' ability to capture the nuanced contextual values inherent to these communities. Both TextBlob and Vader demonstrate poor performance compared to human coding, highlighting the need for improved sentiment analysis techniques tailored to sensitive online communities like #meanspo. Additional limitations occur when media is attached with tweets contributing to the sentiments. This study contextualizes how human involvement and expertise are essential for exploring these communities, as relying solely on automatic classifications can be risky and fail to grasp the complex dynamics and implications of such online interactions. Future contextual work is essential to evaluate the risks and harms of using automatic classification models in sensitive online communities and to develop effective human-centered strategies to mitigate these impacts. TRIGGER WARNING: potentially triggering content.