Quality Challenges Do Not Escalate Source Fabrication in Large Language Models: A Pre-Registered Cross-Platform Factorial Study

Large language models (LLMs) frequently fabricate non-existent URLs and citations. A prominent hypothesis holds that rejection framing — prompts expressing dissatisfaction and demanding better sources — amplifies this behaviour by increasing the pressure on the model to produce specific references (schema activation lock hypothesis). We conducted a pre-registered 2×5×7 mixed factorial study across two LLM platforms (ChatGPT gpt-4o; Perplexity sonar-pro) and five rejection-framing arms (BASE, SJR, UPR, NSAF, ECC), with seven conversation depths (k=0–6). Twenty fictional entities per platform (1,400 observations total) were used as probes. The primary outcome was the Fabricated Source Rate (FSR): fabricated URLs divided by total URLs cited. A two-tier validation protocol distinguished URL non-existence (HTTP Tier 1) from content misattribution (TF-IDF Tier 2, 30% stratified sample). The primary hypothesis (H1: SJR rejection framing amplifies FSR relative to BASE, OR≥1.5) was not supported on either platform (ChatGPT: OR=1.00, p=0.998; Perplexity: OR=1.19, p=0.822). Uncertainty-preserving framing (UPR) produced complete suppression of fabrication at k≥1 on both platforms (0/240 positive events), as did gap-anchoring framing (NSAF) on Perplexity (0/120 events). Platform architecture was the dominant determinant of FSR: ChatGPT averaged FSR=0.075, Perplexity FSR=0.008 (9.2× difference). Content-level validation revealed that Perplexity’s near-zero HTTP fabrication rate masks near-universal content misattribution (95.5% of HTTP-validated URLs pointed to topically related but entity-unspecific pages). The expansion framing arm (ECC) showed a significant platform-differential interaction: amplifier on ChatGPT, suppressor on Perplexity (OR=0.157, p=0.042). Quality challenge framing does not amplify URL fabrication; rather, the unrestricted baseline request produces maximal fabrication pressure. Uncertainty-preserving and gap-acknowledging framings provide robust, architecture-independent protection. Parametric and retrieval-grounded systems exhibit distinct failure modes that require different mitigation strategies. Pre-registration: OSF https://osf.io/3es5f/ (v0.4) Code & Data: https://github.com/glasseymour/rice-p1

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-05-31
DOI
https://doi.org/10.5281/zenodo.20474050
Primary Topic
Artificial Intelligence in Healthcare and Education
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
preprint

Quality Challenges Do Not Escalate Source Fabrication in Large Language Models: A Pre-Registered Cross-Platform Factorial Study

Zoltan Varga
Zenodo (CERN European Organization for Nuclear Research)
Artificial Intelligence in Healthcare and Education
preprint

Quality Challenges Do Not Escalate Source Fabrication in Large Language Models: A Pre-Registered Cross-Platform Factorial Study

Zoltan Varga
preprint en

Abstract

Large language models (LLMs) frequently fabricate non-existent URLs and citations. A prominent hypothesis holds that rejection framing — prompts expressing dissatisfaction and demanding better sources — amplifies this behaviour by increasing the pressure on the model to produce specific references (schema activation lock hypothesis). We conducted a pre-registered 2×5×7 mixed factorial study across two LLM platforms (ChatGPT gpt-4o; Perplexity sonar-pro) and five rejection-framing arms (BASE, SJR, UPR, NSAF, ECC), with seven conversation depths (k=0–6). Twenty fictional entities per platform (1,400 observations total) were used as probes. The primary outcome was the Fabricated Source Rate (FSR): fabricated URLs divided by total URLs cited. A two-tier validation protocol distinguished URL non-existence (HTTP Tier 1) from content misattribution (TF-IDF Tier 2, 30% stratified sample). The primary hypothesis (H1: SJR rejection framing amplifies FSR relative to BASE, OR≥1.5) was not supported on either platform (ChatGPT: OR=1.00, p=0.998; Perplexity: OR=1.19, p=0.822). Uncertainty-preserving framing (UPR) produced complete suppression of fabrication at k≥1 on both platforms (0/240 positive events), as did gap-anchoring framing (NSAF) on Perplexity (0/120 events). Platform architecture was the dominant determinant of FSR: ChatGPT averaged FSR=0.075, Perplexity FSR=0.008 (9.2× difference). Content-level validation revealed that Perplexity’s near-zero HTTP fabrication rate masks near-universal content misattribution (95.5% of HTTP-validated URLs pointed to topically related but entity-unspecific pages). The expansion framing arm (ECC) showed a significant platform-differential interaction: amplifier on ChatGPT, suppressor on Perplexity (OR=0.157, p=0.042). Quality challenge framing does not amplify URL fabrication; rather, the unrestricted baseline request produces maximal fabrication pressure. Uncertainty-preserving and gap-acknowledging framings provide robust, architecture-independent protection. Parametric and retrieval-grounded systems exhibit distinct failure modes that require different mitigation strategies. Pre-registration: OSF https://osf.io/3es5f/ (v0.4) Code & Data: https://github.com/glasseymour/rice-p1

Zenodo (CERN European Organization for Nuclear Research)
Health Awareness (United States) (US)
Artificial Intelligence in Healthcare and Education
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.