Can Language Models Recognize Reused Evidence? Cross-Task Errors and Prompt Sensitivity in Provenance Judgments
Agreement among reports need not imply independent underlying evidence. We evaluate whether four open-weight instruction-tuned models (1.5–3.8B parameters) distinguish established source independence (YES), confirmed reuse (NO), and unresolved provenance (UNCERTAIN), and whether global judgments agree with local reused-pair identification. On a 24-family, 72-case constructed narrative corpus, confirmed-reuse recall under the archived P0 instruction is 0–3.47%, despite higher aggregate and pair-task accuracies. Three frozen prompt variants on the same narratives produce pronounced answer-distribution shifts: Phi-3.5 reaches 75% NO recall with explicit definitions but responds NO on every input when the permitted answer order changes. On a distinct, pre-inference-frozen 36-family structured-register corpus (6,048 GPU responses), global macro-F1 ranges from 0.184 to 0.404; Qwen2.5-1.5B incorrectly asserts independence on 256/432 designated reuse cases, whereas SmolLM2’s 100% reuse recall comes from near-constant NO predictions. Across both constructions, constant-answer baselines, permutation-invariant errors, and global–local disagreement make aggregate accuracy an unreliable stand-alone measure. These are controlled diagnostics against author-assigned labels, not estimates for unrestricted scientific papers. A separately supplied independent single-human-reviewer record covers 96 cases but cannot yet be aligned case by case with the frozen corpora. Retrospective lexical-shortcut susceptibility also limits inference from narrative scores.
Authors
- Manish Singh Bisht
- Uttam Tiwari (ORCID: https://orcid.org/0009-0002-4215-991X)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-19
- DOI
- https://doi.org/10.5281/zenodo.24362261
- Primary Topic
- Topic Modeling
- Type
- preprint