Can Reasoning LLMs Synthesize Complex Climate Statements?
Accurately synthesizing climate evidence into concise statements is crucial for policy making and fostering public trust in climate science.Recent advancements in Large Language Models (LLMs), particularly the emergence of reasoning-optimized variants, which excel at mathematical and logical tasks, present a promising yet untested opportunity for scientific evidence synthesis.We evaluate stateof-the-art reasoning LLMs on two key tasks:(1) contextual confidence classification, assigning appropriate confidence levels to climate statements based on evidence, and (2) factual summarization of climate evidence, generating concise summaries evaluated for coherence, faithfulness, and similarity to expertwritten versions.Using a novel dataset of 612 structured examples constructed from the Sixth Assessment Report (AR6) of the Intergovernmental Panel on Climate Change (IPCC), we find reasoning LLMs outperform generalpurpose models in confidence classification by 8 percentage points in accuracy and macro-F1 scores.However, for summarization tasks, performance differences between model types are mixed.Our findings demonstrate that reasoning LLMs show promise as auxiliary tools for confidence assessment in climate evidence synthesis, while highlighting significant limitations in their direct application to climate evidence summarization.This work establishes a foundation for future research on the targeted integration of LLMs into scientific assessment workflows.Code and
Authors
- Yucheng Lu
Institutions
- New York University (US)
Publication Details
- Published
- 2025-01-01
- DOI
- https://doi.org/10.18653/v1/2025.climatenlp-1.21
- Primary Topic
- Semantic Web and Ontologies
- Type
- article
- Field-Weighted Citation Impact
- 0.00