Patient Asks to AI: Is there Anybody there? Patient-Centered AI Approach and Human Oversight

Patient- centered AI and its design means to AI system that prioritize human needs, values and capabilities. Furthermore, patient-centered AI systems shall be designed to allow for human oversight and control throughout the entire interaction process. Patient values, preferences, and capacity data, when used for patient-centered artificial intelligence systems, will present a high risk. On the one hand, this mechanism, which aims to strengthen the patient's health and prioritize their happiness, will be trained with a large amount of personal data. The data scale is very diverse. For patient preferences, data such as treatment preferences, end-of-life preferences, desire to participate in the decision-making process, confidentiality preferences, moral, religious, financial values, physical and emotional capacity will be used to train the AI system. This study examines patient-centered AI systems through the lens of high-risk artificial intelligence, emphasizing the necessity of a human oversight mechanism to ensure that the patient receives high-quality patient-centered service while the system is supervised by a human, and highlighting the need for human element oversight and supervision in diagnosis and treatment. Within the scope of the EU AI Act specifically Article 14 (Human Oversight) under the high-risk category, the risks facing patients and recommendations will be discussed. It touches on the necessity of the coexistence of the Patient-Centered AI mechanism and the Human Oversight mechanism, and in this context also sheds light on the contradictions between a patient-centered AI system and a human oversight mechanism. In this context, due to the possibility that the patient-centered AI mechanism can be added to medical devices, it is an extension, and it is a part of a software, it has also been evaluated together with the Medical Devices Regulation.