Before Measuring Epistemic Change: The Dependence of Scientific Antecedent Retrieval on Text Representation, Retrieval Models, and Temporal Windows
Purpose. Measures of scientific novelty and epistemic change frequently oper-ationalize a document’s relationship to prior knowledge. Yet the identification of that prior knowledge is itself a measurement operation whose robustness to documentary representation, retrieval architecture, and temporal eligibility is rarely made explicit. This study examines these dependencies rather than treating retrieved antecedents as an invariant reference set. Methods. We investigated a harmonized corpus of 993,086 Brazilian postgraduate theses and dissertations from 2013–2024. A second-stage balanced sample contained 108 focal documents, twelve from each of nine broad disciplinary areas, dated 2016–2024. For each focal, eligible antecedents were earlier documents within the same harmonized specific disciplinary area. We constructed four retrieval configurations: BM25 using titles and keywords or titles, abstracts, and keywords, and two correspond-ing multilingual sentence-embedding representations (M2 and M4). We compared complete available historical windows with rolling three-year windows, assessing Top 10/30/50 overlap, uniform-selection chance references, stratified bootstrap intervals, and semantic-score geometry. A separate year-matched reference design compared same-area and other-area similarities. Results. Across 108 focal docu-ments, the semantic experiment evaluated 1,640,271 focal–antecedent pairs in the historical window and 756,979 in the three-year window. In the historical Top 30, overlap was 20.25% between M2 and M4 and 22.41% between short- and full-text BM25. Agreement between text-aligned lexical and semantic configurations was 29.17% (BM25 short–M2) and 26.98% (BM25 full–M4), compared with 17.07% and 16.20% for crossed-field comparisons. The average paired advantage of text-aligned over crossed comparisons was 11.44 percentage points (95% stratified bootstrap interval, 10.19–12.67). Excluding 2016, for which the available histori-cal and three-year windows coincide by construction, temporal Top-30 stability ranged from 51.31% for short BM25 to 55.28% for M4. Year-matched same-area minus other-area cosine contrasts were 0.01196 for M2 and 0.01425 for M4. Conclusions. Identification of potential scientific antecedents is sensitive to seemingly upstream measurement decisions. The findings establish operational uncertainty in the construction of reference sets for novelty-oriented bibliometrics, but nei-ther demonstrate influence relationships nor validate the epistemic relevance of retrieved documents. Robustness analysis of antecedent retrieval should precede substantive interpretation of similarity-derived evidence of epistemic change.
Authors
- Mardochée Ogecime (ORCID: https://orcid.org/0000-0001-7846-6941)
Institutions
- Universidade Federal de Minas Gerais (BR)
Publication Details
- Journal
- Research Square
- Published
- 2026-11-19
- DOI
- https://doi.org/10.21203/rs.3.rs-11460349/v1
- Primary Topic
- scientometrics and bibliometrics research
- Type
- preprint