Sed quis custodet? Sentencing intentions of judges are more variable than those of lay people.

Background: There is a general assumption that judges assess crimes more reliably than jurors or the general public. Reliability is fundamental for trust and validity. Here, I compare the variability of sentencing intentions of judges and lay people.Quantifying the variability of sentencing is difficult, because sentencing guidelines impose boundaries on sentence lengths, that cause problems for estimation. I compared two ways to circumvent these problems: (1) by analysing actual sentences as proportions of the guideline range, or (2) by assuming that observed sentences reflect a latent distribution of “true” intentions. In both cases, the primary outcome was the variability of judges’ sentences compared with lay people’s.Methods: I analysed published sentencing intentions of professional judges and lay people for the same fictional crime. The source study’s sentencing guidelines allowed terms in the range 3-20 years, and over half the participants’ sentencing intentions were at these boundaries. I accounted for these boundary effects by transforming the raw sentence intentions to either (1) to proportions of the allowed range, or (2) to an ordinal variable, with 4 levels (0-9, 10-14, 15-19 or 20 years), representing a latent normal distribution. I then analysed the transformed data using distributional regressions, to assess ‘dispersion’ (reliability) in relation to rater status (judge, lay person)Results: Accounting for floor and ceiling effects importantly improved estimation of reliability. Judges’ sentencing intentions were more variable than those of lay people in both analyses. The proportional (relative) and ordinal (absolute) methods of analysis estimated lower or higher mean sentences for judges.Conclusion: Distributional regression can assess differential variability, as an indicator of reliability. The present results challenge the assumption that judges assess crimes reliably. This may help to explain the public’s lack of trust in the judicial system. It also throws into question the decisions of single expert raters in other high-stakes contexts.