General scales unlock AI evaluation with explanatory and predictive power
. Here we introduce general scales for AI evaluation that elicit demand profiles explaining what capabilities common AI benchmarks truly measure, extract ability profiles quantifying the general strengths and limits of AI systems and robustly predict AI performance for new task instances. Our fully automated methodology builds on 18 rubrics, capturing a broad range of cognitive and intellectual demands, which place different task instances on the same general scales, illustrated on 15 large language models (LLMs) and 63 tasks. Both the demand and the ability profiles on these scales bring new insights such as construct validity through benchmark sensitivity and specificity and explain conflicting claims about whether AI has reasoning capabilities. Ultimately, high predictive power at the instance level becomes possible using the general scales, providing superior estimates over strong black-box baseline predictors, especially in out-of-distribution settings (new tasks and benchmarks). The scales, rubrics, battery, techniques and results presented here constitute a solid foundation for a science of AI evaluation, underpinning the reliable deployment of AI in the years ahead.
Authors
- David Stillwell (ORCID: https://orcid.org/0000-0003-0174-3212)
- Lucy G. Cheke (ORCID: https://orcid.org/0000-0001-5588-7575)
- Jiyun Zu (ORCID: https://orcid.org/0000-0003-1076-5093)
- Lorenzo Pacchiardi (ORCID: https://orcid.org/0000-0003-4760-7638)
- Zongqian Li
- Jonathan E. Prunty (ORCID: https://orcid.org/0000-0002-9180-1932)
- Katherine M. Collins (ORCID: https://orcid.org/0000-0002-7032-716X)
- Lexin Zhou (ORCID: https://orcid.org/0000-0003-1161-4270)
- Peter Henderson (ORCID: https://orcid.org/0000-0003-3938-0541)
- Patrick C. Kyllonen (ORCID: https://orcid.org/0000-0002-6517-4576)
- Yael Moros-Daval (ORCID: https://orcid.org/0000-0001-5442-2055)
- Luning Sun
- Qinlin Zhao
- Seraphina Zhang (ORCID: https://orcid.org/0009-0009-3587-385X)
- Sherry Tongshuang Wu
- Kexin Jiang-Chen
- José Hernández-Orallo
- Pablo A M Casares
- Fernando Martínez-Plumed
- Yitian Huang
- Behzad Mehrbakhsh
- Pablo Sánchez-García
- John Burden
Institutions
- Educational Testing Service (US)
- William & Mary (US)
- Princeton University (US)
- University of Cambridge (GB)
- Williams (United States) (US)
- Leverhulme Trust (GB)
- Microsoft Research Asia (China) (CN)
- Centre for Automation and Robotics (ES)
- Artificial Intelligence Research Institute (ES)
- Universitat Politècnica de València (ES)
- Carnegie Mellon University (US)
- KU Leuven (BE)
Publication Details
- Journal
- Nature
- Published
- 2026-04-01
- DOI
- https://doi.org/10.1038/s41586-026-10303-2
- Primary Topic
- Explainable Artificial Intelligence (XAI)
- Type
- article
- Field-Weighted Citation Impact
- 0.00