SciWorkBench: A Benchmark for Real Scientific Workflows
Language models are increasingly being used as agents capable of planning and executing projects, as well as refining their actions through feedback. This shift raises a question: can current agents complete projects like real scientists? To examine this question, we introduce SciWorkBench, a benchmark for evaluating agents' abilities on real science tasks across biology, chemistry, materials science, and physics. SciWorkBench is built from peer-reviewed literature or expert-contributed research scenarios. We also propose a unified scientific analysis framework validated by domain experts, which can categorize most tasks in SciWorkBench. Each task covers one or more stages of this framework. We evaluate up-to-date models within the Claude Code and Codex agent frameworks under a unified execution and grading protocol. The best-performing agent achieves a mean score of 57.38, indicating that current agents still struggle with realistic scientific workflows. Our failure analysis identifies three recurring weaknesses at the workflow level: difficulty translating scientific knowledge into task-specific procedures, weak integration of evidence across different modalities, and limited capacity for scientific self-correction. These findings highlight the importance of workflow-based evaluation for measuring progress toward reliable AI systems for scientific research.
Authors
- Qile Jin
- 尹燕妮
- Wai Yuet Chiu (ORCID: https://orcid.org/0000-0003-0552-8026)
- Qingteng Chen
- Jinghang Wang
- Taolin Han
- Lin Qu (ORCID: https://orcid.org/0009-0004-2028-0780)
- Kewei Sun
- Ruodan Chen
- Jialu Nie
- Junchao Li
- Jiaxin Li
- Yuchen Zhang
- Jiajia Li
- Hu Wei
- Yuting Fu
- Jinxin Wang
- Xiaogang Li
- Chen Zhao
- Bing Zhao
- Yifei Zhang
- Yun Wu
Publication Details
- Journal
- arXiv (Cornell University)
- Published
- 2026-11-17
- DOI
- https://doi.org/10.48550/arxiv.2611.22490
- Primary Topic
- Scientific Computing and Data Management
- Type
- preprint