Preventing Visibility and Metadata from Substituting for Evidence in AI-Assisted Scholarship

AI-assisted writing and repository workflows can increase the visibility of educational and scholarly artifacts without increasing support for their claims. This study treats that problem as a decision-rule issue: when evidence, authorization, provenance, and exposure are collapsed into one score, discoverable metadata can compensate for weak evidentiary support. A stochastic model generates synthetic research objects with separate semantic evidence, governance authorization, provenance completeness, and exposure readiness. Three release rules are compared: an additive checklist, an evidence-bound gate, and a provenance-adaptive gate that adds second review for high-consequence or low-provenance objects. Across 40 independent simulations of 20,000 objects, baseline false reassurance among released objects was 60.32 percent for the additive checklist, 30.41 percent for the evidence-bound gate, and 23.02 percent for the adaptive gate. Increasing exposure readiness alone raised additive release coverage from 6.96 percent to 16.02 percent and false reassurance to 72.80 percent; the gated rules were unchanged by design. Inflating recorded evidence for weak objects degraded every rule, with the adaptive gate reaching 58.76 percent false reassurance. The model also reports release coverage, sensitivity, specificity, balanced accuracy, and review burden. It is an executable evaluation template, not a validation of TRACE-AI, an estimate of real publication error, or evidence for adopting a deployed workflow.

Authors

Institutions

Publication Details

Published
2026-11-02
DOI
https://doi.org/10.31222/osf.io/4vsny_v1
Primary Topic
Publishing and Scholarly Communication
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
preprint

Preventing Visibility and Metadata from Substituting for Evidence in AI-Assisted Scholarship

Lauren Mercer, Maya Kalyani Reed, Hoang Justin Tran, Alexander J. Garnier
Publishing and Scholarly Communication
preprint

Preventing Visibility and Metadata from Substituting for Evidence in AI-Assisted Scholarship

Lauren Mercer, Maya Kalyani Reed, Hoang Justin Tran, Alexander J. Garnier
preprint en

Abstract

AI-assisted writing and repository workflows can increase the visibility of educational and scholarly artifacts without increasing support for their claims. This study treats that problem as a decision-rule issue: when evidence, authorization, provenance, and exposure are collapsed into one score, discoverable metadata can compensate for weak evidentiary support. A stochastic model generates synthetic research objects with separate semantic evidence, governance authorization, provenance completeness, and exposure readiness. Three release rules are compared: an additive checklist, an evidence-bound gate, and a provenance-adaptive gate that adds second review for high-consequence or low-provenance objects. Across 40 independent simulations of 20,000 objects, baseline false reassurance among released objects was 60.32 percent for the additive checklist, 30.41 percent for the evidence-bound gate, and 23.02 percent for the adaptive gate. Increasing exposure readiness alone raised additive release coverage from 6.96 percent to 16.02 percent and false reassurance to 72.80 percent; the gated rules were unchanged by design. Inflating recorded evidence for weak objects degraded every rule, with the adaptive gate reaching 58.76 percent false reassurance. The model also reports release coverage, sensitivity, specificity, balanced accuracy, and review burden. It is an executable evaluation template, not a validation of TRACE-AI, an estimate of real publication error, or evidence for adopting a deployed workflow.

Simon Fraser University (CA), Concordia University (CA), University of Missouri (US), Université Grenoble Alpes (FR)
Publishing and Scholarly Communication
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon