A framework for evaluating the chemical knowledge and reasoning abilities of large language models against the expertise of chemists

Large language models (LLMs) have gained widespread interest owing to their ability to process human language and perform tasks on which they have not been explicitly trained. However, we possess only a limited systematic understanding of the chemical capabilities of LLMs, which would be required to improve models and mitigate potential harm. Here we introduce ChemBench, an automated framework for evaluating the chemical knowledge and reasoning abilities of state-of-the-art LLMs against the expertise of chemists. We curated more than 2,700 question-answer pairs, evaluated leading open- and closed-source LLMs and found that the best models, on average, outperformed the best human chemists in our study. However, the models struggle with some basic tasks and provide overconfident predictions. These findings reveal LLMs' impressive chemical capabilities while emphasizing the need for further research to improve their safety and usefulness. They also suggest adapting chemistry education and show the value of benchmarking frameworks for evaluating LLMs in specific domains.

Authors

Institutions

Publication Details

Journal
Nature Chemistry
Published
2025-05-20
DOI
https://doi.org/10.1038/s41557-025-01815-x
Citations
40
Primary Topic
Machine Learning in Materials Science
Type
article
Field-Weighted Citation Impact
20.71

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

A framework for evaluating the chemical knowledge and reasoning abilities of large language models against the expertise of chemists

A.H. Mirza, Lea C. Klepsch, Michael Ringleb, M.V. Gil et al.
40 citations
Nature Chemistry
Machine Learning in Materials Science
20.71
article

A framework for evaluating the chemical knowledge and reasoning abilities of large language models against the expertise of chemists

A.H. Mirza, Lea C. Klepsch, Michael Ringleb, M.V. Gil, Martiño Ríos-García, Santiago Miret, Mehrdad Asgari, Amir Mohammad Elahi, Kevin Maik Jablonka, Philippe Schwaller, Michael Pieler, Fabian Alexander Kreth, Ashraf S. Ibrahim, Nicole C. Roesner, Jakob Meyer, Hani M. Elbeheiry, Christina Glaubitz, T. Gupta, Jan Matthias Peschel, Caroline T. Holick, Ulrich S. Schubert, J. Eberhardt, A. D. Dinga Wonanke, Maximilian Greiner, Leanne M. Stafast, Yannik Köster, Sreekanth Kunchapu, Benedict Emoekabu, Macjonathan Okereke, Nawaf Alampara, Aswanth Krishnan, Tim Hoffmann, J. Schreiber, Mara Schilling-Wilhelmi, Anagha Aneesh
article en
40 citations

Abstract

Large language models (LLMs) have gained widespread interest owing to their ability to process human language and perform tasks on which they have not been explicitly trained. However, we possess only a limited systematic understanding of the chemical capabilities of LLMs, which would be required to improve models and mitigate potential harm. Here we introduce ChemBench, an automated framework for evaluating the chemical knowledge and reasoning abilities of state-of-the-art LLMs against the expertise of chemists. We curated more than 2,700 question-answer pairs, evaluated leading open- and closed-source LLMs and found that the best models, on average, outperformed the best human chemists in our study. However, the models struggle with some basic tasks and provide overconfident predictions. These findings reveal LLMs' impressive chemical capabilities while emphasizing the need for further research to improve their safety and usefulness. They also suggest adapting chemistry education and show the value of benchmarking frameworks for evaluating LLMs in specific domains.

Nature ChemistryVol. 17(7)
Helmholtz Institute Jena (DE), Intel (United States) (US), University of Cambridge (GB), Open Geospatial Consortium (GB), Sustainability Institute (ZA), Instituto Nacional del Carbón (ES), AM Technologies (Poland) (PL), École Polytechnique Fédérale de Lausanne (CH), University of Bayreuth (DE), Friedrich Schiller University Jena (DE), Technische Universität Dresden (DE)
National Science Foundation, U.S. Department of State, Friedrich-Schiller-Universität Jena, Government of the United Kingdom, UK Research and Innovation, Ministerio de Ciencia, Innovación y Universidades, US-UK Fulbright Commission, European Commission, Deutsche Forschungsgemeinschaft, Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung, Office of Multicultural Interests Department of Local Government and Communities, Helmholtz Association, Slezská Univerzita v Opavě, HORIZON EUROPE Framework Programme, Consejo Superior de Investigaciones Científicas, Southeastern Ontario Academic Medical Organization, Agencia Estatal de Investigación
Quality Education
Openalex Percentile: Top 0%
Machine Learning in Materials Science
20.71
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.