LLMs as jurors: assessing the potential of Large Language Models in legal contexts

Objectives: We explored the potential of Large Language Models (LLMs) in legal decision-making by replicating Fraser et al.'s (2023) mock jury experiment using LLMs (GPT-4o, Claude 3.5 Sonnet, and GPT-o1) as decision-makers. We investigated LLMs' reactions to factors that influenced human jurors, including defendant race, social status, number of allegations, and reporting delay in sexual assault cases. Hypotheses: We hypothesized that LLMs would show higher consistency than humans, with no explicit but potential implicit biases. We also examined potential mediating factors (race-crime congruence, credibility, Black Sheep Effect) and moderating effects (beliefs about traumatic memory, ease of reporting) explaining LLM decision-making. Methods: Using a 2x2x2x3 factorial design, we manipulated defendant race (Black/White), social status (low/high), number of allegations (one/five), and reporting delay (5/20/35 years), collecting 2304 responses across conditions. LLMs were prompted to act as jurors, providing probability of guilt assessments (0-100), dichotomous verdicts, and responses to mediator and moderator variables. Results: LLMs showed higher average probability of guilt assessments compared to humans (63.56 vs. 58.82) but were more conservative in rendering guilty verdicts (21% vs. 49%). Similar to humans, LLMs demonstrated bias against White defendants and increased guilt attributions with multiple allegations. Unlike humans, who showed minimal effects of reporting delay, LLMs assigned higher guilt probabilities to cases with shorter reporting delays. Mediation analyses revealed that race-crime stereotype congruency and the Black Sheep Effect partially mediated the racial bias effect, while perceived memory strength mediated the reporting delay effect. Conclusion: While LLMs may offer more consistent decision-makings, they are not immune to biases and may interpret certain case factors differently than human jurors.