Can LLM Self-Verification Detect Factual Errors? A Small Empirical Study

Large Language Models (LLMs) are increasingly used to generate and interpret information, but their ability to provide factually accurate responses remains a significant challenge. This study investigates whether providing explicit reference evidence improves the accuracy of factual claim classification by a large language model. A small-scale comparative experiment was conducted using GPT-6 through ChatGPT and a dataset of 20 factual claims, primarily related to astronomy and planetary science. The claims were evaluated under two experimental conditions. In Condition A, the model classified claims without supplied reference evidence, while in Condition B, the same claims were presented with verified reference evidence. The model's predictions were compared against independently established gold labels. The results showed that both conditions achieved an accuracy of 95%, with 19 correct predictions out of 20. Although the overall accuracy remained unchanged, two individual claims produced different predictions between the conditions. These findings indicate that providing reference evidence can influence a model's factual judgements but does not necessarily guarantee improved accuracy. The study provides a small-scale exploratory assessment of evidence-assisted factual classification and highlights the need for further research involving larger datasets, multiple language models and automated evidence retrieval.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-19
DOI
https://doi.org/10.5281/zenodo.24369816
Primary Topic
Topic Modeling
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Can LLM Self-Verification Detect Factual Errors? A Small Empirical Study

Aarya Apte
Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
preprint

Can LLM Self-Verification Detect Factual Errors? A Small Empirical Study

Aarya Apte
preprint en

Abstract

Large Language Models (LLMs) are increasingly used to generate and interpret information, but their ability to provide factually accurate responses remains a significant challenge. This study investigates whether providing explicit reference evidence improves the accuracy of factual claim classification by a large language model. A small-scale comparative experiment was conducted using GPT-6 through ChatGPT and a dataset of 20 factual claims, primarily related to astronomy and planetary science. The claims were evaluated under two experimental conditions. In Condition A, the model classified claims without supplied reference evidence, while in Condition B, the same claims were presented with verified reference evidence. The model's predictions were compared against independently established gold labels. The results showed that both conditions achieved an accuracy of 95%, with 19 correct predictions out of 20. Although the overall accuracy remained unchanged, two individual claims produced different predictions between the conditions. These findings indicate that providing reference evidence can influence a model's factual judgements but does not necessarily guarantee improved accuracy. The study provides a small-scale exploratory assessment of evidence-assisted factual classification and highlights the need for further research involving larger datasets, multiple language models and automated evidence retrieval.

Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon