Deterministic Evidence Retrieval vs Generative LLMs: A Comparative Audit of Citation Fabrication in Sports & Exercise Medicine

Three-arm comparative audit quantifying scientific citation fabrication across contemporary large language models (LLMs) in sports and exercise medicine, with a deterministic retrieval engine (EvidenX) as control arm. Twelve published systematic reviews (2019–2025) served as gold standards (940 references via Europe PMC). Five LLMs were tested in memory mode (web search/RAG disabled) across three conditions: unaided generation, deterministic retrieval, and LLM grounded on verified references. Pooled citation fabrication in unaided generation was 49.1% (613 citations; 95% CI 45.2–53.1), ranging from 33.9% (Grok-4.3) to 84.1% (Groq Llama-3.3). A frozen verifier with deterministic title-fallback correction revealed that naïve identifier-only verification over-counts fabrication by 17.5 percentage points — a systematic super-counting bias affecting prior identifier-based audits. Fabrication rose monotonically with topic niche-ness. An expanded replication (v2, 2026-06-13) on a disjoint set of 42 PICO questions across 9 subdomains and 6 models reproduced the pooled rate at 48.0% (564/1176 citations; 95% CI 45.1–50.8) and refined the subdomain gradient from supplements/ergogenics (38.2%) to concussion/neurology (55.6%). Deterministic retrieval (EvidenX) produced 0% fabrication by design; grounding LLMs on verified references eliminated added fabrication across all models, though evidence utilization varied two-fold and independently of standalone accuracy. Blind external validation by three independent reviewers confirmed the primary fabrication endpoint (Fleiss' κ=0.855; human–machine κ=0.95). This preprint is a pilot / proof-of-concept, JMIR-structured. Full v2 tables and provenance in V2_RESULTS.md.