Regulating AI-Driven Triage: Fundamental Rights and Compliance Challenges in the European Union

Emergency triage is a critical healthcare action that could be improved through the use of artificial intelligence (AI) systems, as these have been shown to achieve accuracy rates of approximately 70–90% for LLMs and AUC values ranging from 0.75 to 0.95 for common AI models. However, these systems face challenges related to the rights and interests of the individuals involved. The European Union’s normative framework, including not only data protection regulations but also the AI Act and medical device regulations, imposes conditions on the use of AI, and these are analyzed here. Our conclusions reveal that Article 22 of the General Data Protection Regulation (GDPR) makes it difficult to justify the establishment of fully automated decision-making models for triage. That accountability obligations for implementers (Fundamental Rights Impact Assessments: FRIAs) and data controllers (data protection impact assessments: DPIAs) can contribute to better design of AI-based decision-making in triage. Furthermore, with regard to the information rights set out in the GDPR, these have been complemented by the right to an explanation under Art. 86 AI Act in the use of high-risk AI systems. Unfortunately, regulation relating to general-purpose AI models may create some gaps in this framework. The implementation of AI systems for automated decision-making in triage has the potential to improve medical care, but their use requires clarification of applicable regulations and safeguards for patients’ rights.