Assessment of fine-tuned large language models for real-world chemistry and material science applications

The current generation of large language models (LLMs) has limited chemical knowledge. Recently, it has been shown that these LLMs can learn and predict chemical properties through fine-tuning. Using natural language to train machine learning models opens doors to a wider chemical audience, as field-specific featurization techniques can be omitted. In this work, we explore the potential and limitations of this approach. We studied the performance of fine-tuning three open-source LLMs (GPT-J-6B, Llama-3.1-8B, and Mistral-7B) for a range of different chemical questions. We benchmark their performances against "traditional" machine learning models and find that, in most cases, the fine-tuning approach is superior for a simple classification problem. Depending on the size of the dataset and the type of questions, we also successfully address more sophisticated problems. The most important conclusions of this work are that, for all datasets considered, their conversion into an LLM fine-tuning training set is straightforward and that fine-tuning with even relatively small datasets leads to predictive models. These results suggest that the systematic use of LLMs to guide experiments and simulations will be a powerful technique in any research study, significantly reducing unnecessary experiments or computations.

Authors

Institutions

Publication Details

Journal
Chemical Science
Published
2024-11-22
DOI
https://doi.org/10.1039/d4sc04401k
Citations
42
Primary Topic
Machine Learning in Materials Science
Type
article
Field-Weighted Citation Impact
4.92

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Assessment of fine-tuned large language models for real-world chemistry and material science applications

K. J. Schmidt, Guillaume Godin, Andreas Züttel, Clémence Corminbœuf et al.
42 citations
Chemical Science
Machine Learning in Materials Science
4.92
article

Assessment of fine-tuned large language models for real-world chemistry and material science applications

K. J. Schmidt, Guillaume Godin, Andreas Züttel, Clémence Corminbœuf, Elena Simone, Ruaraidh D. McIntosh, M.V. Gil, Hilal Daglar, Jonathan K. Whitmer, Xin Jin, Sauradeep Majumdar, D. C. Florian Wieland, Mehrdad Asgari, Amir Mohammad Elahi, Kevin Maik Jablonka, Beatriz Mouriño, Ian Foster, Matthew Garvin, Ben Blaiszik, Andy S. Anker, D. Tolnai, Tuomas P. J. Knowles, Antonio Buffo, Susana García, Kadi L. Saar, Regine Willumeit‐Römer, Rubén Laplaza, Seyed Mohamad Moosavi, Jiale Shi, Joren Van Herck, Tanja Junkers, Igor V. Tetko, Berend Smit, Lydia L. Good, Hossein Mashhadimoslem, Marco Taddei, Dennis Svatunek, Fahimeh Hooriabad Saboor, Sahar Vahdatifar, C. Pevida, Morteza Sagharichiha, Seda Keskın, Francesca Nerli, Leander Choudhury, Alex Abrudan, N.‐X. Hu, Mahyar Rajabi Kochi, Michele Lessona, Neda Poudineh, Jianan Gu
article en
42 citations

Abstract

The current generation of large language models (LLMs) has limited chemical knowledge. Recently, it has been shown that these LLMs can learn and predict chemical properties through fine-tuning. Using natural language to train machine learning models opens doors to a wider chemical audience, as field-specific featurization techniques can be omitted. In this work, we explore the potential and limitations of this approach. We studied the performance of fine-tuning three open-source LLMs (GPT-J-6B, Llama-3.1-8B, and Mistral-7B) for a range of different chemical questions. We benchmark their performances against "traditional" machine learning models and find that, in most cases, the fine-tuning approach is superior for a simple classification problem. Depending on the size of the dataset and the type of questions, we also successfully address more sophisticated problems. The most important conclusions of this work are that, for all datasets considered, their conversion into an LLM fine-tuning training set is straightforward and that fine-tuning with even relatively small datasets leads to predictive models. These results suggest that the systematic use of LLMs to guide experiments and simulations will be a powerful technique in any research study, significantly reducing unnecessary experiments or computations.

Chemical ScienceVol. 16(2)
University of Notre Dame (US), University of Pisa (IT), Argonne National Laboratory (US), Helmholtz Institute Jena (DE), National Institutes of Health (US), Koç University (TR), TU Wien (AT), University of Waterloo (CA), Politecnico di Torino (IT), University of Toronto (CA), University of Tehran (IR), University of Cambridge (GB), Helmholtz Zentrum München (DE), Heriot-Watt University (GB), University of Oxford (GB), University of Chicago (US), National Institute of Diabetes and Digestive and Kidney Diseases (US), Instituto Nacional del Carbón (ES), Helmholtz-Zentrum Hereon (DE), Baxter (Germany) (DE), École Polytechnique Fédérale de Lausanne (CH), Monash University (AU), Massachusetts Institute of Technology (US), University of Mohaghegh Ardabili (IR), Friedrich Schiller University Jena (DE), Technical University of Denmark (DK)
National Science Foundation, Carl-Zeiss-Stiftung, Frances and Augustus Newman Foundation, Grantham Foundation for the Protection of the Environment, Government of the United Kingdom, UK Research and Innovation, European Commission, Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung, Cambridge Trust, University of Toronto, Novo Nordisk, Ministerio de Ciencia e Innovación, Novo Nordisk Fonden, National Institutes of Health, Seventh Framework Programme, HORIZON EUROPE Framework Programme, European Research Council, Consejo Superior de Investigaciones Científicas, European Regional Development Fund, National Center of Competence in Research Materials’ Revolution: Computational Design and Discovery of Novel Materials, Agencia Estatal de Investigación, National Institute of Diabetes and Digestive and Kidney Diseases, H2020 European Research Council, H2020 Marie Skłodowska-Curie Actions
Quality Education
Openalex Percentile: Top 2%
Machine Learning in Materials Science
4.92
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.