Closed Corpora, Open Questions: Rethinking Statistical Evidence in Computational Humanities

Computational humanities research has begun to inherit the evidentiary standards of the empirical sciences. For an important class of such work, I argue, this expectation is misplaced. Null-hypothesis significance testing (NHST), a cornerstone of quantitative research, supports inference from a sample to a larger population. However, much computational research in the humanities works with closed corpora, such as the complete works of an author or every episode of a long-running television series. Such corpora are not independent samples from a specifiable population. P-values computed on them lack their intended meaning. We still want findings to matter beyond the corpus, and they can, but not through statistical arguments. In this paper, I propose a two-stage division of labor. Computational claims are made as exact statements about the corpus in hand, and extension beyond it proceeds through qualitative interpretation using the methods of generalization the humanities already possess. In place of asking whether observed differences would be unlikely under a null hypothesis, the framework asks how separable the categories of interest are in the space of chosen metrics, measured with supervised classifiers under leave-one-out cross-validation. The approach is illustrated with a case study of stylistic distinctiveness across six national variants of the game show Taskmaster, using machine-learning-derived metrics of visual and aural style, and two findings are then extended beyond the corpus interpretively. The paper closes with a reflection on implications for reproducibility and replicability in computational humanities.