Auditing Dataset Shortcuts in Fake News Detection and Scoring Manipulative Writing Style with a Local Browser Extension
Text classifiers for fake news report accuracy above 0.99 on the ISOT corpus. We show that much of that figure comes from how easily the corpus can be recognized: the single word "reuters", used as a presence rule, labels 99.4% of it correctly, because real articles carry wire datelines and fake ones do not. We remove this and related artifacts with a cleaning module shared by training and serving, retrain a TF-IDF logistic regression model, and repeat the evaluation on a second corpus of 52,722 articles from eight sources. Five-fold accuracy falls from 0.9855 to 0.9736 on ISOT and from 0.9304 to 0.8959 on the second corpus. Under topic and source shifts the cleaned models reach 0.92 in one direction and 0.52 to 0.66 in the other three, below the majority-class rate in two of them. Because the classifier does not transfer, we pair it with ten interpretable stylometric signals and report a manipulation score from 0 to 100, half from the classifier and half from the signals that fire. The system is a Manifest V3 Chrome extension that calls a local FastAPI service, so article text stays on the machine; the server needs a median of 0.8 to 4.5 ms per request. Worked examples separate a clickbait text (score 82) from a wire-style report (23) and expose two failures: a sober false story scores as low as a true one, and a neutral explainer receives a moderate score. The tool makes no claim about factual accuracy.
Authors
- Abhinav
- Rajvir Singh
Institutions
- Sharda University (IN)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-17
- DOI
- https://doi.org/10.5281/zenodo.24290954
- Primary Topic
- Misinformation and Its Impacts
- Type
- preprint