How Chemical Evidence Shapes Mechanistic Reasoning and Prediction by Large Language Models: Fine-Grained Inference in Nucleoside-Derived Supramolecular Gelation

Large language models (LLMs) offer a distinctive capability for chemical problem solving by integrating chemical knowledge and experimental evidence through reasoning expressed in language. Yet how chemical evidence reshapes their predictions and expressed rationales is not self-evident and has received limited examination, particularly for challenging chemical problems. Here, using nucleoside-derived supramolecular gelation as a testbed, we systematically varied the evidence provided to LLMs and quantitatively evaluated their predictions and the mechanistic alignment of generated rationales. Without additional evidence, correct predictions were associated with greater alignment to literature-supported mechanisms than incorrect ones, suggesting that LLMs can draw on intrinsic chemical knowledge despite rationales remain largely general. Structurally similar gelation precedents, thought improved prediction accuracy significantly, reduced this distinction by anchoring rationales to insufficiently discriminative structural features, indicating that similarity-based evidence does not necessarily improve chemical reasoning and that prediction accuracy alone is insufficient to assess LLM performance in chemical tasks. In contrast, condition-constrained references and mechanistic information more consistently directed rationales toward chemically discriminative features, even when prediction accuracy did not improve in tandem. Together, our study highlights the importance of task-specific evidence design and quantitative rationale evaluation for reliable chemical problem solving by LLMs.