From Data to Verified Model Knowledge: How Evidence Access Shapes Large Language Model Answers About Trained Machine-Learning Classifiers

Large language models (LLMs) are increasingly used to answer questions about predictive models, but a fluent answer about a model is not necessarily true of it. We propose a framework in which every answer is checked against a reference that independent software computes from the fitted model. The main finding is simple: the LLM was much more accurate when software had done the calculations than when it had to work them out from the model itself. In a confirmatory study with Gemini 2.5 Flash, the LLM correctly answered 16 of 36 questions whose answer follows exactly from the full model description, such as a prediction or a decision path, but 33 of the same 36 once software had computed the values from which the answer follows. A replication with 50 classifiers, five datasets and three LLM configurations, including a reasoning setting and an open-weight model, reproduced this pattern. Without model information, the LLMs declined to answer or guessed at chance level, and verified answers were reported faithfully once their origin was stated. Model-specific calculations should therefore be left to verified software, while the LLM reports and explains the checked results.

Authors

Institutions

Publication Details

Journal
Information
Published
2026-11-19
DOI
https://doi.org/10.3390/info17111179
Primary Topic
Explainable Artificial Intelligence (XAI)
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

From Data to Verified Model Knowledge: How Evidence Access Shapes Large Language Model Answers About Trained Machine-Learning Classifiers

Attila Ficsor, János Abonyi
Information
Explainable Artificial Intelligence (XAI)
article

From Data to Verified Model Knowledge: How Evidence Access Shapes Large Language Model Answers About Trained Machine-Learning Classifiers

Attila Ficsor, János Abonyi
article en

Abstract

Large language models (LLMs) are increasingly used to answer questions about predictive models, but a fluent answer about a model is not necessarily true of it. We propose a framework in which every answer is checked against a reference that independent software computes from the fitted model. The main finding is simple: the LLM was much more accurate when software had done the calculations than when it had to work them out from the model itself. In a confirmatory study with Gemini 2.5 Flash, the LLM correctly answered 16 of 36 questions whose answer follows exactly from the full model description, such as a prediction or a decision path, but 33 of the same 36 once software had computed the values from which the answer follows. A replication with 50 classifiers, five datasets and three LLM configurations, including a reasoning setting and an open-weight model, reproduced this pattern. Without model information, the LLMs declined to answer or guessed at chance level, and verified answers were reported faithfully once their origin was stated. Model-specific calculations should therefore be left to verified software, while the LLM reports and explains the checked results.

InformationVol. 17(11)
University of Pannonia (HU), Hungarian Research Network (HU)
Explainable Artificial Intelligence (XAI)
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon