From Data to Verified Model Knowledge: How Evidence Access Shapes Large Language Model Answers About Trained Machine-Learning Classifiers
Large language models (LLMs) are increasingly used to answer questions about predictive models, but a fluent answer about a model is not necessarily true of it. We propose a framework in which every answer is checked against a reference that independent software computes from the fitted model. The main finding is simple: the LLM was much more accurate when software had done the calculations than when it had to work them out from the model itself. In a confirmatory study with Gemini 2.5 Flash, the LLM correctly answered 16 of 36 questions whose answer follows exactly from the full model description, such as a prediction or a decision path, but 33 of the same 36 once software had computed the values from which the answer follows. A replication with 50 classifiers, five datasets and three LLM configurations, including a reasoning setting and an open-weight model, reproduced this pattern. Without model information, the LLMs declined to answer or guessed at chance level, and verified answers were reported faithfully once their origin was stated. Model-specific calculations should therefore be left to verified software, while the LLM reports and explains the checked results.
Authors
- Attila Ficsor (ORCID: https://orcid.org/0000-0002-0541-4590)
- János Abonyi (ORCID: https://orcid.org/0000-0001-8593-1493)
Institutions
- University of Pannonia (HU)
- Hungarian Research Network (HU)
Publication Details
- Journal
- Information
- Published
- 2026-11-19
- DOI
- https://doi.org/10.3390/info17111179
- Primary Topic
- Explainable Artificial Intelligence (XAI)
- Type
- article
- Field-Weighted Citation Impact
- 0.00