Challenges for human oversight in detecting system malfunctions in AI-assisted medical diagnostics
Increasingly, physicians must oversee AI systems to detect system malfunctions. We studied 48 physicians in a randomized controlled study using an increasingly malfunctioning AI for cardiac auscultation. Diagnostic performance declined alongside AI reliability, even when physicians were cognitively forced to make an initial classification, and when human performance exceeded AI reliability. Physicians' confidence declined faster than their trust in AI. Results call for specification of oversight goals and processes ensuring patient safety.
Authors
- Niels van Berkel (ORCID: https://orcid.org/0000-0001-5106-7692)
- Nadine Schlicker (ORCID: https://orcid.org/0000-0003-0368-3081)
- Markus Langer (ORCID: https://orcid.org/0000-0002-8165-1803)
- Ivica Grgic (ORCID: https://orcid.org/0000-0001-9017-1804)
- Martin Christian Hirsch (ORCID: https://orcid.org/0000-0003-3268-4921)
- Andreas J. Mayr (ORCID: https://orcid.org/0000-0001-7106-9732)
- Ali Kholmovaia
- Noëlle Steffens
- Constanze Tepohl
Institutions
- University of Freiburg (DE)
- Philipps University of Marburg (DE)
- Technical University of Munich (DE)
- Aalborg University (DK)
Publication Details
- Journal
- npj Digital Medicine
- Published
- 2026-11-13
- DOI
- https://doi.org/10.1038/s41746-026-03410-x
- Primary Topic
- Artificial Intelligence in Healthcare and Education
- Type
- article
- Field-Weighted Citation Impact
- 0.00
Funders
- Deutsche Forschungsgemeinschaft