Artificial Intelligence Snake Oil in Medicine

In the midst of the current hype of artificial intelligence (AI) in almost all aspects or our life, two scholars in their book, “AI Snake Oil – What Artificial Intelligence Can Do, What it Can’t, and How to Tell the Difference,” offer a nuanced perspective.[1] In their own words, they do not hesitate “to call a spade a spade and a sham a sham.” Some excerpts from their illuminating book which has relevance in medicine are discussed in the following paragraphs. But they caution against sweeping generalizations. One cannot deny that AI applications have improved rapidly. Ipso facto this has driven the hype which has been so misleading at times. AI is not a single technology. Its usefulness varies a lot driven by different applications, different tasks, and the context. AI is poised to play a big role in healthcare. We should proceed cautiously because of past misadventures with hurriedly introduced AI tools without validating them using real-world data. Many AI promoters pushed the concept that using electronic health records, hospitals can use AI to predict sepsis. This would help in introducing early treatment and prevent deaths. A US-based company named Epic had the largest collection of electronic health records consisting of over 250 million people.[2] Using this database, Epic launched an AI product to detect sepsis in 2017. Hospitals could use this tool with their existing health records to reduce mortality from sepsis. The AI developer marketed their tool with claims that many hospitals had used their system with good results. In an interview in 2020, Judith Faulkner, the CEO, stated, “If you put in sepsis AI…six hours before the human being can tell that this patient is coming up with sepsis, it can identify sepsis in many cases and save lives.”[1] On the pretext of proprietary trade secret, the company did not offer any peer-reviewed study supporting this claim. The product was launched without independent researchers who could replicate the results claimed by Epic. Its use continued for 4 years before researchers from the University of Michigan Medical School carried out an independent evaluation in a hospital in which they worked.[3] The findings of this study were disappointing. The “AI snake oil” had claimed that its model had an accuracy between 76 and 83%. But the investigators found that the accuracy was around 63%, or little more than tossing a coin which would give 50% accuracy. Experienced clinicians would perform far better than this. Following this in October 2022, Epic withdrew its sepsis prediction model. But the harm to patients during the 4 years the model was in use would has been considerable. This misadventure with the AI sepsis model should be taken as a caution by hospital administrators considering deploying AI tools in patient care. They should look for peer-reviewed independent studies before investing in AI technology. At the individual level also, there have been reports of harms to mental health from AI tools. We are facing a pandemic of loneliness. This has driven many to take recourse to companion chatbots which purportedly provide emotional support as claimed by many users.[2] Driving the hype, a Google engineer, Blake Lemoine, in June 2022 declared that the company’s chatbot had become sentient, that is, able to feel.[4] Thankfully, Google did not concur and Lemoine was sent on administrative leave. Nevertheless, not to be left behind in the hype race, Bing’s chatbot claimed to be sentient in early 2023.[1] This supplement reports an interesting communication along with a commentary on how such hypes can have an adverse impact at the individual level in the form of emerging phenomena like “AI psychosis.” Author contributions AB was responsible for all work related to the editorial, such as concept, writing, drafting, revision and final approval. Financial support and sponsorship Nil. Conflicts of interest Nil. Data availability statement Not available.