Fake News Detection Under Latent Semantic Domain Shift

High benchmark performance in fake news detection may reflect interpolation within familiar semantic distributions rather than robustness to latent semantic domain shift. However, datasets containing veracity labels without explicit semantic domain annotations prevent direct assessment of cross-domain generalization. To address this limitation, we propose an unsupervised domain-discovery framework that infers latent semantic partitions, enabling systematic evaluation across distinct semantic domains. Using 127,173 tweets from TruthSeeker 2023, domains were inferred from Sentence-BERT embeddings through K-Means, Agglomerative Clustering, HDBSCAN, and BERTopic. Four classifiers (LR, SVM, BERT, and RoBERTa) were evaluated under conventional random splitting, statement-grouped splitting, within-domain testing, pairwise cross-domain transfer, Leave-One-Domain-Out (LODO), and domain-balanced LODO. Conventional tweet-level evaluation achieved accuracy above 0.98, whereas statement-grouped splitting reduced accuracy to 0.77–0.83, suggesting potential inflation from claim-level overlap. Pairwise transfer revealed substantial degradation, with Macro-F1 ranging from 0.28 to 0.54 across LR-informed challenging pairs (SSD). Performance-independent randomly sampled pairs (RSD) yielded higher Macro-F1 (0.34–0.70), while still exhibiting considerable degradation. In contrast, mean LODO Macro-F1 ranged from 0.80 to 0.96, whereas domain balancing provided no systematic improvement. Exploratory structural analyses on a limited common domain subset suggested modest associations between semantic or lexical separation and transfer degradation. Overall, the findings highlight the importance of unsupervised domain discovery for evaluating robustness when domain annotations are unavailable and demonstrate the need for complementary claim-disjoint, single-source, and multi-source generalization assessments.