Signal Certification in Digital Humanities: An Empirically Calibrated Framework for Distinguishing Statistical Signal from Methodological Artifact

Signal Certification in Digital Humanities: An Empirically Calibrated Framework for Distinguishing Statistical Signal from Methodological Artifact The growing use of computational methods and AI-assisted research pipelines in the digital humanities has increased the capacity to generate candidate patterns in cultural heritage data, while raising a fundamental methodological question: how can researchers determine whether an observed signal reflects a genuine property of a corpus rather than an artifact of analytical choices? This paper presents the Artifact Probability Score (APS), a five-component framework designed to assess the robustness of computationally detected signals against specification sensitivity, compositional confounding, resampling instability, multiple-comparison fragility, and subgroup non-replication. The framework is empirically calibrated using fourteen reference corpora and controls spanning eight languages and four script families. A complementary sensitivity analysis evaluates the stability of APS classifications under systematic perturbations of the component weights. The results reveal both the framework’s potential and its limitations. The calibration supports a robust classification zone for APS scores between 0 and 20, but shows that the intermediate range does not reliably distinguish genuine compositional effects from ordinary variation in aggregate lexical structure. Accordingly, the original three-zone classification is revised to two evidentially stronger extremes—Robust (0–20) and Near-certain artifact (76–100)—separated by a broad Uncertain zone (21–75). The sensitivity analysis further indicates that classification instability is concentrated around thresholds rather than in the relative ranking of reference cases. The study provides a reproducible procedure for documenting how much of a candidate signal survives explicit methodological stress tests. Its conclusions remain deliberately bounded: the calibration concerns pairwise correlations between derived numerical quantities across two-way corpus partitions and does not establish the validity of APS for other signal families or research domains. Domain-specific recalibration against known-null and known-structured reference cases is therefore required before transfer. By shifting the methodological emphasis from pattern discovery to signal certification, this work contributes an empirically grounded approach to statistical accountability in digital humanities and computational cultural heritage research.