Lost in the PDFs: Context Degradation and Parametric Hallucinations in LLM UIs

This record contains the preprint paper "LOST IN THE PDFS: CONTEXT DEGRADATION AND PARAMETRIC HALLUCINATIONS IN LLM UIS", authored by Gabriele D’Iorio (University of Naples Parthenope). The study investigates the structural limitations and operational vulnerabilities of commercial web-based Large Language Model (LLM) interfaces when handling long, multi-document sequential workflows in educational contexts. Abstract:The growing use of LLMs for educational purposes, especially by inexperienced users, inevitably leads to LLMs hallucinating and making mistakes. This study presents an experiment using the web version of Gemini (Pro Standard and Flash Standard modes) to simulate a numerical analysis class covering three topics (interpolation, piecewise interpolation, and approximation), each in a separate PDF. This study is subdivided into two scenarios created to evaluate how the LLM responded and, above all, to see how long it took to hallucinate or suffer from context degradation. The results demonstrate the failure of both scenarios due to context degradation. Despite large theoretical context windows, sudden attachment truncation in the web UI forces the model to rely on its parametric memory, leading to severe hallucinations and sycophantic behavior. Key Topics Covered:- Context Degradation & "Lost in the Middle" Phenomenon: Analysis of how LLMs lose retrieval accuracy in prolonged sessions.- Parametric Hallucinations: How models compensate for context loss by fabricating details from pre-training data.- Document Parsing Challenges: Visual layout issues in standard Web UIs.- Experimental Evaluation: A comparative analysis of a single-chat setup (Scenario A) versus a dual-agent/tutor setup (Scenario B). Data Availability:The comprehensive dataset, interaction logs, prompt configurations, and Python scripts used for token counting are openly accessible in the project’s GitHub repository at: https://github.com/g-projectech/lost_in_the_pdfs/