Social media polarization during conflict: insights from an ideological stance dataset on Reddit comments

In politically sensitive scenarios such as wars, social media often becomes a platform for polarized discourse and expressions of strong ideological stances. While prior studies have explored ideological stance detection in general contexts, limited attention has been given to conflict-specific settings. This study addresses this gap by analyzing 9969 Reddit comments related to the Israel–Palestine conflict collected between October 2023 and August 2024. The comments were manually annotated into three stance classes—Pro-Israel, Pro-Palestine, and Neutral—with high inter-annotator agreement (Fleiss’ κ = 0.93). We conduct a comprehensive comparative evaluation of neural networks, pre-trained language models, and large language models (LLMs), along with multiple prompt engineering strategies and an LLM majority voting ensemble. Model performance is assessed using accuracy, precision, recall, and F1-score. Among the tested approaches, the Scoring and Reflective Re-read prompting strategy applied to Mixtral 8x7B achieves the best overall performance across evaluation metrics. Beyond model performance, the findings provide empirical insights into how ideological polarization manifests in online discourse surrounding the Israel–Palestine conflict, offering a computational perspective on the distribution of Pro-Israel, Pro-Palestine, and Neutral narratives. The dataset used in this study is publicly available to support further research on ideological stance detection in politically sensitive contexts.