Towards a Metrological Validation of PFAS Property Predictions by In-Context LLMs Across Four Architectures

In recent years, the use of large-language models (LLMs) as prediction tools has become increasingly popular. Despite this popularity, the metrological validation of LLMs in the study of chemical problems has been somehow neglected to date. To illustrate this need in chemistry, we introduce a minimal, metrologically grounded framework for validating in-context learning (ICL)-based prediction in LLMs on molecular physicochemical properties with reliable estimation of uncertainty. Our framework integrates a multicycle testing protocol (Mc = 50 replicate cycles per model-property cell) with a six-metric bundle: central tendency (mean MAE over replicate cycles), reproducibility (interquartile range), regression performance (RMSE, and R2), information-theoretical lower bound on the amount of captured signal (𝐼" in nats), and ordinal 2 performance (accuracy against the median in the dataset), plus three cross-property and crossmodel comparison metrics: NRMSE, Welch’s z-test, and Scaling Law Fit exponents. We use this framework to validate four LLMs, two dense (8B and 70B parameters; 1B = 109) and two mixture-of-experts (MoE; 17B active parameters with 109B and 400B in total), for predicting eight per- and polyfluoroalkyl substance (PFAS) properties: boiling point (BP), melting point (MP), vapor pressure (VP), Henry’s law constant (HLC), octanol-water partition coefficient (LogKow), aqueous solubility (LogS), acid dissociation constant (pKa), and critical micelle concentration (LogCMC). In this benchmark, the frontier MoE model (Maverick 400B parameters) achieves a mean validation NRMSE of 14.4 %, and the mid-scale MoE model (Scout, 109B parameters) outperforms our larger dense model (Llama-3.3-70B) despite having 4.1× fewer active parameters. This suggests that MoE routed capacity can substitute for activeparameter scale in chemical applications of ICL. When compared to published QSPR and DFT baselines, LLM-ICL beats these approaches on three of eight directly comparable properties (pKa by 43 %, LogKow by 31 %, HLC by 10 %) and trails behind on five (BP, MP, LogS, VP by 12 % to 75 %, and LogCMC by a factor of approx. 3.3), despite using approximately 32 incontext examples (32 for seven properties, 20 for HLC) versus the training sets comprising 102 to 105 molecules used in these approaches. We also discuss a validation-set anomaly in the case of LogKow, where our mid-scale dense model Llama-3.3-70B outperforms the frontier MoE model on all four validation metrics, possibly due to better coverage of the pretraining data. Permutation (shuffled-label) and information-access blinding controls show that genuine structure-response signal is present, but that the native-prompt accuracy depends substantially on pretrained, property-specific knowledge and familiar target scales. Overall, our results suggest a two-fold conclusion on the competitiveness of LLM-ICL: it succeeds in predicting chemical

Authors

Institutions

Publication Details

Journal
ChemRxiv
Published
2026-11-19
DOI
https://doi.org/10.26434/chemrxiv.15012473/v1
Primary Topic
Machine Learning in Materials Science
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Towards a Metrological Validation of PFAS Property Predictions by In-Context LLMs Across Four Architectures

Carlos A. Gonzalez
ChemRxiv
Machine Learning in Materials Science
preprint

Towards a Metrological Validation of PFAS Property Predictions by In-Context LLMs Across Four Architectures

Carlos A. Gonzalez
preprint en

Abstract

In recent years, the use of large-language models (LLMs) as prediction tools has become increasingly popular. Despite this popularity, the metrological validation of LLMs in the study of chemical problems has been somehow neglected to date. To illustrate this need in chemistry, we introduce a minimal, metrologically grounded framework for validating in-context learning (ICL)-based prediction in LLMs on molecular physicochemical properties with reliable estimation of uncertainty. Our framework integrates a multicycle testing protocol (Mc = 50 replicate cycles per model-property cell) with a six-metric bundle: central tendency (mean MAE over replicate cycles), reproducibility (interquartile range), regression performance (RMSE, and R2), information-theoretical lower bound on the amount of captured signal (𝐼" in nats), and ordinal 2 performance (accuracy against the median in the dataset), plus three cross-property and crossmodel comparison metrics: NRMSE, Welch’s z-test, and Scaling Law Fit exponents. We use this framework to validate four LLMs, two dense (8B and 70B parameters; 1B = 109) and two mixture-of-experts (MoE; 17B active parameters with 109B and 400B in total), for predicting eight per- and polyfluoroalkyl substance (PFAS) properties: boiling point (BP), melting point (MP), vapor pressure (VP), Henry’s law constant (HLC), octanol-water partition coefficient (LogKow), aqueous solubility (LogS), acid dissociation constant (pKa), and critical micelle concentration (LogCMC). In this benchmark, the frontier MoE model (Maverick 400B parameters) achieves a mean validation NRMSE of 14.4 %, and the mid-scale MoE model (Scout, 109B parameters) outperforms our larger dense model (Llama-3.3-70B) despite having 4.1× fewer active parameters. This suggests that MoE routed capacity can substitute for activeparameter scale in chemical applications of ICL. When compared to published QSPR and DFT baselines, LLM-ICL beats these approaches on three of eight directly comparable properties (pKa by 43 %, LogKow by 31 %, HLC by 10 %) and trails behind on five (BP, MP, LogS, VP by 12 % to 75 %, and LogCMC by a factor of approx. 3.3), despite using approximately 32 incontext examples (32 for seven properties, 20 for HLC) versus the training sets comprising 102 to 105 molecules used in these approaches. We also discuss a validation-set anomaly in the case of LogKow, where our mid-scale dense model Llama-3.3-70B outperforms the frontier MoE model on all four validation metrics, possibly due to better coverage of the pretraining data. Permutation (shuffled-label) and information-access blinding controls show that genuine structure-response signal is present, but that the native-prompt accuracy depends substantially on pretrained, property-specific knowledge and familiar target scales. Overall, our results suggest a two-fold conclusion on the competitiveness of LLM-ICL: it succeeds in predicting chemical

ChemRxiv
National Institute of Standards and Technology (US), Material Measurement Laboratory (US)
Machine Learning in Materials Science
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon