LLMs Are Not Oracles: Knowledge Graphs Map Logical Fallacies in GenAI Math Explanations Across AI Assistants and Human Shadows

Large language models (LLMs) are increasingly used in math education, yet little is known about how persona conditioning reshapes LLM math explanations. We examine 28,000 runs from the Math Education Digital Shadows (MEDS) dataset, spanning 14 LLMs as AI assistants (i.e., vanilla AI) or as human shadows (i.e., personifications). We analyze qualitative mathematics-related responses through textual forma mentis networks, knowledge graphs including word–word syntactic, semantic and emotional associations. We also use a DistilBERT-based classifier to flag five categories of potential logical fallacies. A regex-based check of the PCA responses showed that reassuring language was frequent but model-dependent and was not always flagged as appeal to emotion. Persona conditioning produced model-dependent changes in network size, density, and algebraic connectivity. Aggregated human-shadow responses displayed broader lexical coverage. Answers to easy and medium math questions often had no classifier-flagged fallacies, whereas an anxiety-framed PCA prompt sharply increased appeal-to-emotion and relevance labels, especially in AI-assistant mode. On an 18-item quiz, AI assistants’ accuracy exceeded human shadows’ in 10 out of 14 architectures. Expressed confidence exceeded empirical accuracy in 13 of 14 AI-assistant and 11 of 14 human-shadow conditions, a descriptive gap, not a calibration measure. These results show that (i) cognitive digital shadowing modulates LLM-generated mathematical discourse compared to the vanilla AI assistant case, and (ii) the tested LLMs are not infallible mathematics oracles. No measure here directly assesses the explanations’ mathematical validity or teaching quality.

Authors

Institutions

Publication Details

Journal
Computers
Published
2026-11-18
DOI
https://doi.org/10.3390/computers15110849
Primary Topic
Intelligent Tutoring Systems and Adaptive Learning
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

LLMs Are Not Oracles: Knowledge Graphs Map Logical Fallacies in GenAI Math Explanations Across AI Assistants and Human Shadows

Ali Aghazadeh Ardebili, Massimo Stella, NAOMI ESPOSITO
Computers
Intelligent Tutoring Systems and Adaptive Learning
article

LLMs Are Not Oracles: Knowledge Graphs Map Logical Fallacies in GenAI Math Explanations Across AI Assistants and Human Shadows

Ali Aghazadeh Ardebili, Massimo Stella, NAOMI ESPOSITO
article en

Abstract

Large language models (LLMs) are increasingly used in math education, yet little is known about how persona conditioning reshapes LLM math explanations. We examine 28,000 runs from the Math Education Digital Shadows (MEDS) dataset, spanning 14 LLMs as AI assistants (i.e., vanilla AI) or as human shadows (i.e., personifications). We analyze qualitative mathematics-related responses through textual forma mentis networks, knowledge graphs including word–word syntactic, semantic and emotional associations. We also use a DistilBERT-based classifier to flag five categories of potential logical fallacies. A regex-based check of the PCA responses showed that reassuring language was frequent but model-dependent and was not always flagged as appeal to emotion. Persona conditioning produced model-dependent changes in network size, density, and algebraic connectivity. Aggregated human-shadow responses displayed broader lexical coverage. Answers to easy and medium math questions often had no classifier-flagged fallacies, whereas an anxiety-framed PCA prompt sharply increased appeal-to-emotion and relevance labels, especially in AI-assistant mode. On an 18-item quiz, AI assistants’ accuracy exceeded human shadows’ in 10 out of 14 architectures. Expressed confidence exceeded empirical accuracy in 13 of 14 AI-assistant and 11 of 14 human-shadow conditions, a descriptive gap, not a calibration measure. These results show that (i) cognitive digital shadowing modulates LLM-generated mathematical discourse compared to the vanilla AI assistant case, and (ii) the tested LLMs are not infallible mathematics oracles. No measure here directly assesses the explanations’ mathematical validity or teaching quality.

ComputersVol. 15(11)
University of Trento (IT), Center for Neuroscience and Cognitive Systems (IT)
Intelligent Tutoring Systems and Adaptive Learning
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon