Benchmarking LLM abilities on postdoc-level natural science problems
Scientific problems have no answer key, so evaluating an agent means deciding what counts as a better solution before any agent is run. Existing benchmarks either match a symbolic answer against a key or, where they require executable work, score it by unit tests, by a rubric, or by position on a competition leaderboard. None of these is the criterion by which a field decides that one published method beats another. STEMBench adopts that criterion directly. Each of its twelve research problems in high-energy physics, astrophysics and neuroscience is scored by its own field's figure of merit, and each comes with two measured anchors: a submission that uses no domain knowledge, and an expert reference reproducing a published method, both obtained by running code shipped with the benchmark. The interval between them states how much of a problem is crossed by scientific insight rather than by engineering. Across eight frontier models, task-level wins and domain-level leadership dissociate: a model that wins five of the twelve tasks leads no domain mean. In the high-effort cross-model comparison, at least one model's mean score exceeds the expert anchor on seven of twelve tasks. The remaining five span uncertainty estimation, latent and causal inference, and cross-site clinical classification. Numerical scores and official-submission validity capture different aspects of performance and are reported separately. We release the tasks, graders, anchors and harness.
Authors
- Irina Piontkovskaya (ORCID: https://orcid.org/0009-0003-0299-5849)
- Е. В. Трофимова (ORCID: https://orcid.org/0000-0001-5436-8511)
- Liqun Deng (ORCID: https://orcid.org/0009-0006-7242-0794)
- D. Derkach
- Mary Godunova
- Dmitry Ilvovsky
- Fedor Ratnikov
- Alina Avanesyan (ORCID: https://orcid.org/0009-0000-0199-2812)
- Zhenhe Zhang
- Mikhail Hushchyn
- Dmitry Polubotko
- Alexey Boldyrev
- Irina Knyazeva
Publication Details
- Journal
- arXiv (Cornell University)
- Published
- 2026-11-18
- DOI
- https://doi.org/10.48550/arxiv.2611.24321
- Primary Topic
- Topic Modeling
- Type
- preprint