Before Measuring Epistemic Change: The Dependence of Scientific Antecedent Retrieval on Text Representation, Retrieval Models, and Temporal Windows

Purpose. Measures of scientific novelty and epistemic change frequently oper-ationalize a document’s relationship to prior knowledge. Yet the identification of that prior knowledge is itself a measurement operation whose robustness to documentary representation, retrieval architecture, and temporal eligibility is rarely made explicit. This study examines these dependencies rather than treating retrieved antecedents as an invariant reference set. Methods. We investigated a harmonized corpus of 993,086 Brazilian postgraduate theses and dissertations from 2013–2024. A second-stage balanced sample contained 108 focal documents, twelve from each of nine broad disciplinary areas, dated 2016–2024. For each focal, eligible antecedents were earlier documents within the same harmonized specific disciplinary area. We constructed four retrieval configurations: BM25 using titles and keywords or titles, abstracts, and keywords, and two correspond-ing multilingual sentence-embedding representations (M2 and M4). We compared complete available historical windows with rolling three-year windows, assessing Top 10/30/50 overlap, uniform-selection chance references, stratified bootstrap intervals, and semantic-score geometry. A separate year-matched reference design compared same-area and other-area similarities. Results. Across 108 focal docu-ments, the semantic experiment evaluated 1,640,271 focal–antecedent pairs in the historical window and 756,979 in the three-year window. In the historical Top 30, overlap was 20.25% between M2 and M4 and 22.41% between short- and full-text BM25. Agreement between text-aligned lexical and semantic configurations was 29.17% (BM25 short–M2) and 26.98% (BM25 full–M4), compared with 17.07% and 16.20% for crossed-field comparisons. The average paired advantage of text-aligned over crossed comparisons was 11.44 percentage points (95% stratified bootstrap interval, 10.19–12.67). Excluding 2016, for which the available histori-cal and three-year windows coincide by construction, temporal Top-30 stability ranged from 51.31% for short BM25 to 55.28% for M4. Year-matched same-area minus other-area cosine contrasts were 0.01196 for M2 and 0.01425 for M4. Conclusions. Identification of potential scientific antecedents is sensitive to seemingly upstream measurement decisions. The findings establish operational uncertainty in the construction of reference sets for novelty-oriented bibliometrics, but nei-ther demonstrate influence relationships nor validate the epistemic relevance of retrieved documents. Robustness analysis of antecedent retrieval should precede substantive interpretation of similarity-derived evidence of epistemic change.

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-19
DOI
https://doi.org/10.21203/rs.3.rs-11460349/v1
Primary Topic
scientometrics and bibliometrics research
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Before Measuring Epistemic Change: The Dependence of Scientific Antecedent Retrieval on Text Representation, Retrieval Models, and Temporal Windows

Mardochée Ogecime
Research Square
scientometrics and bibliometrics research
preprint

Before Measuring Epistemic Change: The Dependence of Scientific Antecedent Retrieval on Text Representation, Retrieval Models, and Temporal Windows

Mardochée Ogecime
preprint en

Abstract

Purpose. Measures of scientific novelty and epistemic change frequently oper-ationalize a document’s relationship to prior knowledge. Yet the identification of that prior knowledge is itself a measurement operation whose robustness to documentary representation, retrieval architecture, and temporal eligibility is rarely made explicit. This study examines these dependencies rather than treating retrieved antecedents as an invariant reference set. Methods. We investigated a harmonized corpus of 993,086 Brazilian postgraduate theses and dissertations from 2013–2024. A second-stage balanced sample contained 108 focal documents, twelve from each of nine broad disciplinary areas, dated 2016–2024. For each focal, eligible antecedents were earlier documents within the same harmonized specific disciplinary area. We constructed four retrieval configurations: BM25 using titles and keywords or titles, abstracts, and keywords, and two correspond-ing multilingual sentence-embedding representations (M2 and M4). We compared complete available historical windows with rolling three-year windows, assessing Top 10/30/50 overlap, uniform-selection chance references, stratified bootstrap intervals, and semantic-score geometry. A separate year-matched reference design compared same-area and other-area similarities. Results. Across 108 focal docu-ments, the semantic experiment evaluated 1,640,271 focal–antecedent pairs in the historical window and 756,979 in the three-year window. In the historical Top 30, overlap was 20.25% between M2 and M4 and 22.41% between short- and full-text BM25. Agreement between text-aligned lexical and semantic configurations was 29.17% (BM25 short–M2) and 26.98% (BM25 full–M4), compared with 17.07% and 16.20% for crossed-field comparisons. The average paired advantage of text-aligned over crossed comparisons was 11.44 percentage points (95% stratified bootstrap interval, 10.19–12.67). Excluding 2016, for which the available histori-cal and three-year windows coincide by construction, temporal Top-30 stability ranged from 51.31% for short BM25 to 55.28% for M4. Year-matched same-area minus other-area cosine contrasts were 0.01196 for M2 and 0.01425 for M4. Conclusions. Identification of potential scientific antecedents is sensitive to seemingly upstream measurement decisions. The findings establish operational uncertainty in the construction of reference sets for novelty-oriented bibliometrics, but nei-ther demonstrate influence relationships nor validate the epistemic relevance of retrieved documents. Robustness analysis of antecedent retrieval should precede substantive interpretation of similarity-derived evidence of epistemic change.

Research Square
Universidade Federal de Minas Gerais (BR)
scientometrics and bibliometrics research
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon