Auditing Archive-Size Effects in Transformer-Based Semantic Monitoring of AI Research Literature

Field-level semantic distinctiveness can be estimated by measuring the semantic distance between a focal paper and previous papers. The methodological background comes from previous research on transformer-based document representation, semantic novelty, and topic dynamics, and the challenges of longitudinal nearest-neighbor monitoring arise from the growth of archives, the choice of representation, and the definition of the neighborhood. This study audits these sources of sensitivity in historical semantic distinctiveness. We establish that, for nested prior-paper sets, adding candidates cannot increase the mean distance to the k nearest prior papers. The audit is based on the title-abstract text of a text-audited, relevance-ranked OpenAlex sample of 5,429 artificial intelligence (AI)-related records from 2018–2025. MiniLM, SciBERT, SPECTER and past-only term frequency–inverse document frequency (TF-IDF) representations are evaluated using exact cosine k-nearest-neighbor search (k=5; k=10 sensitivity) under cumulative memory, and MiniLM is also tested under three-year rolling and fixed-capacity memory, while BERTopic-based topic drift and retrieval provenance are explored as a form of exploratory context. The primary comparison is for 2020–2025. Under cumulative memory, endpoint distinctiveness changed by −12.85% for MiniLM, −8.11% for SciBERT, −22.09% for SPECTER, and −2.27% for TF-IDF. A fixed capacity MiniLM memory, by contrast, resulted in a +9.84% endpoint change. This reversal is considered a sensitivity finding, not evidence of a trend in creativity, research quality, or the whole AI field. The study demonstrates that archive dependence needs to be audited prior to the interpretation of semantic distinctiveness as novelty.

Authors

Institutions

Publication Details

Journal
Science Engineering and Technology
Published
2026-11-13
DOI
https://doi.org/10.54327/set2027/v7.i1.421
Primary Topic
scientometrics and bibliometrics research
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Auditing Archive-Size Effects in Transformer-Based Semantic Monitoring of AI Research Literature

Hanaa Mahmood, Omar Alniemi, Alaa Mohsin Saadoon, Saif Alhusseny
4 links
Science Engineering and Technology
scientometrics and bibliometrics research
article

Auditing Archive-Size Effects in Transformer-Based Semantic Monitoring of AI Research Literature

Hanaa Mahmood, Omar Alniemi, Alaa Mohsin Saadoon, Saif Alhusseny
article en

Abstract

Field-level semantic distinctiveness can be estimated by measuring the semantic distance between a focal paper and previous papers. The methodological background comes from previous research on transformer-based document representation, semantic novelty, and topic dynamics, and the challenges of longitudinal nearest-neighbor monitoring arise from the growth of archives, the choice of representation, and the definition of the neighborhood. This study audits these sources of sensitivity in historical semantic distinctiveness. We establish that, for nested prior-paper sets, adding candidates cannot increase the mean distance to the k nearest prior papers. The audit is based on the title-abstract text of a text-audited, relevance-ranked OpenAlex sample of 5,429 artificial intelligence (AI)-related records from 2018–2025. MiniLM, SciBERT, SPECTER and past-only term frequency–inverse document frequency (TF-IDF) representations are evaluated using exact cosine k-nearest-neighbor search (k=5; k=10 sensitivity) under cumulative memory, and MiniLM is also tested under three-year rolling and fixed-capacity memory, while BERTopic-based topic drift and retrieval provenance are explored as a form of exploratory context. The primary comparison is for 2020–2025. Under cumulative memory, endpoint distinctiveness changed by −12.85% for MiniLM, −8.11% for SciBERT, −22.09% for SPECTER, and −2.27% for TF-IDF. A fixed capacity MiniLM memory, by contrast, resulted in a +9.84% endpoint change. This reversal is considered a sensitivity finding, not evidence of a trend in creativity, research quality, or the whole AI field. The study demonstrates that archive dependence needs to be audited prior to the interpretation of semantic distinctiveness as novelty.

Science Engineering and TechnologyVol. 7(1)
University of Mosul (IQ)
scientometrics and bibliometrics research
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon