The Same Corpus, Two Conclusions: How Analyst Choices Can Determine Cross-Platform Findings

Cross-platform web analytics compares sources differing radically in document length, editorial form, and corpus assembly. We show that four routine analyst choices—the unit at which sub-document predictions are aggregated, the summary measure, the relevance criterion, and the order in which documents reach the topic model—can change substantive conclusions about platforms, the three of them decisively and the fourth only where the disagreeing measures do not both separate the pair, while conventional robustness checks detect them only in part. On Ukraine-related discourse from The Guardian, Reddit, and five military OSINT Telegram channels, argmax aggregation took the newspaper’s negative share from 59.3% at sentence level to 90.7% at item level—a movement produced by length, not content, and therefore an error for any comparison meant to be independent of verbosity; a length-invariant measure removed the apparent lead, and which source then looks most negative depends on whose lengths are imposed. Differing relevance criteria reversed which platforms appeared most similar; across twenty-two document orderings, that reversal held in direction but not magnitude. We report and retract the finding in full. The one within-source result we expected to survive, tested blind against pre-specified criteria, is carried by one Telegram channel and reversed in another. No claim about the platforms survives. Several failures share a structure: a figure aggregating heterogeneous units describes none of them. We close with reporting requirements for cross-platform work.