Mitigating Polarisation in Online Discussions Through Adaptive Moderation Techniques

Online discussion moderation/facilitation is crucial for discussions to flourish and prevent polarization and toxicity, which nowadays seems omnipresent. However, because it is heavily based on humans, moderation/facilitation proves costly, time-consuming and non-scalable, which has led many to turn to LLMs for discourse facilitation. In this thesis, we explore the use of LLMs as pseudo-users in online discussions, as a cost-efficient, realistic and scalable way of substituting initial LLM facilitation experiments, which would ordinarily necessitate costly human involvement. Furthermore, we show that including socio-demographic backgrounds in our LLM users leads to more realistic discussions. We explore the use of LLM annotators to estimate discussion quality, using a new statistical test to gauge annotator polarization, and show that using socio-demographic backgrounds in LLM annotators may not meaningfully affect their judgments. Finally, we release a synthetic-discussion creation and annotation framework, the synthetic datasets resulting from our experiments, as well as subsequent analysis and findings from these datasets. Code, datasets and analysis can be found at https://github.com/dimits-ts/llm_moderation_research.