Human and AI Alignment on Stance Detection

Media reporting and public opinion polls following the assassination of UnitedHealthcare CEO Brian Thompson suggested a surprising degree of support for the suspect Luigi Mangione, lack of empathy for the victim, and antipathy towards the health insurance industry. The goal of our project is to examine the social media discourse following the assassination to see whether the mainstream media reporting on this event was accurate and to examine key themes and conflicts in public discourse that emerged on social media. This poster reports the preliminary findings from the stance detection task using Large Language Models (LLMs) and human annotation. We report X users’ stance on Luigi Mangione (In-Favor, Neutral, and Against), and provide interpretation on “In-Favor” and “Against” stances. Further, we report evaluation results on human-human agreement, and human-AI agreement. Our findings and discussion contribute to developing better prompt design for fine-tuning and also guide social scientists in adopting LLMs for stance detection using social media data.