Accuracy Paradox in Large Language Models: Examining Current Regulatory Approaches for Addressing Hallucination

The rapid development and deployment of generative AI systems, particularly large language models (LLMs), have transformed the way information is generated and consumed. As these models become increasingly integrated into critical domains such as healthcare, education, and law, their outputs carry significant epistemic, legal, and societal implications. One of the common regulatory approaches, recognised by both academia and regulatory authorities, is to prioritise accuracy as a primary metric for assessing the reliability of AI-generated content. However, this article argues that such an accuracy-centric approach fails to address deeper challenges and, paradoxically, can exacerbate existing issues. This article conceptualises the accuracy paradox in LLMs, highlighting how improvements in accuracy may enhance the articulation and persuasiveness of outputs while amplifying problems such as hallucination, misinformation, lack of trustworthiness, and the erosion of meaningful transparency. Beyond these concerns, the accuracy-centric approach also risks fostering AI manipulation, nudging behaviours, and marginalising plurality in social and informational contexts. Key values such as group privacy, social heterogeneity, and decision-making autonomy will be overlooked, while entrenched biases and inequalities are inadvertently reinforced. The article critiques current regulatory frameworks of overreliance on statistical accuracy to tackle this issue, ignoring the recursive risks of hallucination loops, where misinformation is perpetuated and worsens over time through training on synthetic data that crowding out human information. It also highlights the dangers of social deskilling, as increasing reliance on AI erodes critical human capacities for judgement and decision-making. Within this conceptual framework, it critically examines the current legal framework of the General Data Protection Law, AI Act and Digital Service Act. It found that these legal frameworks, while addressing certain aspects of aforementioned issues, often fall short in tackling the deeper epistemic and societal challenges posed by hallucination caused risks in LLMs. Specifically, they focus heavily on data accuracy and compliance requirements but fail to adequately account for issues in epistemic and societal levels. To address these challenges, the article calls for a shift in regulatory focus from a accuracy-centric approach to trustworthiness, meaningful transparency, and the recognition of a plurality of values in addressing issues caused by hallucinations in LLMs. This includes mechanisms for reasoning traceability, explicit uncertainty communication, and safeguarding diverse social and informational contexts. By reconceptualising the regulatory landscape, this article aims to develop a more holistic framework for regulating the unique risks posed by GenAI.