Large Language Model Benchmarks as Measurement Systems: Psychometric Contributions and Eight Open Problems

Large language model (LLM) benchmarks compare systems, track progress, and support capability and safety claims, but their scores arise from measurement systems that combine construct definitions, sampled tasks, administration conditions, scoring rules, and intended uses. This methodological article critically reviews how validity theory, generalizability theory, item response theory, adaptive testing, dimensionality analysis, differential item functioning, fit diagnostics, and rater models have informed LLM evaluation. It develops eight nonexhaustive open problems for psychometricians and measurement scientists. The central problem is transposed geometry, where items outnumber independent systems (n ≪ p) More items increase information about system performance over a task population, yet do not add observations for any fixed item. Consequently, parameter recovery, uncertainty quantification, and small-group comparisons may remain unstable without defensible structure. Additional problems concern response-model selection, asymptotes, ordinal and partial-credit outcomes, multidimensionality, dependence and contamination, complex responses, selective administration, temporal drift, linking, and decision uncertainty in practical uses.