An Emergent Understanding of Human-AI Collaboration in Deliberation

Understanding how well large language models (LLMs) can reason deliberatively compared to humans is critical for designing effective human-AI collaboration technologies for deliberation, especially in contexts such as citizen assemblies where participants make policy recommendations to governments. Yet, while existing benchmarks focus on measuring objective accuracy in LLMs’ responses, we know little about how state-of-the-art LLMs compare to human deliberative reasoning. Building on theories for measuring deliberative reasoning from human deliberations, we collected survey data from 54 LLMs and compared them to 526 human responses across 24 deliberation cases to answer the following question: To what extent do LLMs reason deliberatively compared to human participants? Our preliminary findings indicate that humans outperform most LLMs in most deliberation cases, but some LLMs perform on par with humans. These findings suggest that while LLMs are not yet ready to replicate human reasoning in deliberation, their potential as augmentative or representative agents deserves further investigation.