Auditing Dataset Shortcuts in Fake News Detection and Scoring Manipulative Writing Style with a Local Browser Extension

Text classifiers for fake news report accuracy above 0.99 on the ISOT corpus. We show that much of that figure comes from how easily the corpus can be recognized: the single word "reuters", used as a presence rule, labels 99.4% of it correctly, because real articles carry wire datelines and fake ones do not. We remove this and related artifacts with a cleaning module shared by training and serving, retrain a TF-IDF logistic regression model, and repeat the evaluation on a second corpus of 52,722 articles from eight sources. Five-fold accuracy falls from 0.9855 to 0.9736 on ISOT and from 0.9304 to 0.8959 on the second corpus. Under topic and source shifts the cleaned models reach 0.92 in one direction and 0.52 to 0.66 in the other three, below the majority-class rate in two of them. Because the classifier does not transfer, we pair it with ten interpretable stylometric signals and report a manipulation score from 0 to 100, half from the classifier and half from the signals that fire. The system is a Manifest V3 Chrome extension that calls a local FastAPI service, so article text stays on the machine; the server needs a median of 0.8 to 4.5 ms per request. Worked examples separate a clickbait text (score 82) from a wire-style report (23) and expose two failures: a sober false story scores as low as a true one, and a neutral explainer receives a moderate score. The tool makes no claim about factual accuracy.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-17
DOI
https://doi.org/10.5281/zenodo.24290954
Primary Topic
Misinformation and Its Impacts
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Auditing Dataset Shortcuts in Fake News Detection and Scoring Manipulative Writing Style with a Local Browser Extension

Abhinav, Rajvir Singh
4 links
Zenodo (CERN European Organization for Nuclear Research)
Misinformation and Its Impacts
preprint

Auditing Dataset Shortcuts in Fake News Detection and Scoring Manipulative Writing Style with a Local Browser Extension

Abhinav, Rajvir Singh
preprint en

Abstract

Text classifiers for fake news report accuracy above 0.99 on the ISOT corpus. We show that much of that figure comes from how easily the corpus can be recognized: the single word "reuters", used as a presence rule, labels 99.4% of it correctly, because real articles carry wire datelines and fake ones do not. We remove this and related artifacts with a cleaning module shared by training and serving, retrain a TF-IDF logistic regression model, and repeat the evaluation on a second corpus of 52,722 articles from eight sources. Five-fold accuracy falls from 0.9855 to 0.9736 on ISOT and from 0.9304 to 0.8959 on the second corpus. Under topic and source shifts the cleaned models reach 0.92 in one direction and 0.52 to 0.66 in the other three, below the majority-class rate in two of them. Because the classifier does not transfer, we pair it with ten interpretable stylometric signals and report a manipulation score from 0 to 100, half from the classifier and half from the signals that fire. The system is a Manifest V3 Chrome extension that calls a local FastAPI service, so article text stays on the machine; the server needs a median of 0.8 to 4.5 ms per request. Worked examples separate a clickbait text (score 82) from a wire-style report (23) and expose two failures: a sober false story scores as low as a true one, and a neutral explainer receives a moderate score. The tool makes no claim about factual accuracy.

Zenodo (CERN European Organization for Nuclear Research)
Sharda University (IN)
Misinformation and Its Impacts
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon