No One Gets an A: Distributional Bias in LLM Text Scoring

Large language models (LLMs) are increasingly used to rate text quality, yet their scores often cluster within a narrow part of the available scale. We study this score compression through roughly 99,000 evaluations across eleven models. Expert-authored articles are progressively degraded in grammar, coherence, information content, and lexical sophistication, providing a controlled reference gradient. All models compress scores relative to this reference, although the degree varies widely and larger open-weight models generally compress less. Several models never use the scale endpoints, even when simulations predict many such ratings. Mixed-effects analysis confirms that responses to degradation differ across axes and models. We also compare five mitigation strategies. Supervised regression provides the largest improvement in pairwise accuracy, probability-weighted scoring improves rank correlation, and quantile normalization expands the scoring range while preserving rankings. Anchor prompting modestly improves ranking on the matched subset. A simple affine calibration reduces scoring error by 9–26%, but does not recover all lost discrimination. These findings identify a systematic, partly correctable distortion in LLM-generated quality ratings and show why raw scores require calibration before they are used for evaluation or threshold decisions. Code: https://github.com/idreesaziz/Distributional-Bias-in-LLM-Text-Scoring