Off‐The‐Shelf Large Language Models Are Unreliable Judges

ABSTRACT I conduct large‐scale empirical experiments to test the reliability of large language models (LLMs) as legal interpreters. Combining computational methods with the results of a new survey, I find that LLM judgments are highly sensitive to prompt phrasing, output processing methods, and choice of model. I also find that frontier LLMs do not accurately assess linguistic ordinary meaning, and I provide original evidence that tuning base models into assistant models shifts ordinary‐meaning judgments unpredictably and can worsen their alignment with human judgments. These findings undermine LLMs' credibility as legal interpreters and cast doubt on claims that LLMs elucidate ordinary meaning.