39 Can AI explain surgical infection conditions better than surgical residents? A qualitative pilot study

Abstract Introduction Artificial Intelligence (AI) chat bots (CB) are a new tool that can solve and manage various problems, and have applications in Medicine. Currently, many publications are emerging in the medical field exploring how IA can help in clinical practice, with decision-making, medical writing and creating specific content. Few studies explore what uses of IA could be useful to surgical trainees during residency. Surgical infections are one of the most common problems that surgical trainees have to face since they start training. They have to diagnose conditions like appendicitis, diverticulitis and cholecystitis in the Emergency Department (ED) and inform patients about the condition and treatment. One of the most important things requested by patients is accurate information regarding surgery and its complications. In the ED, generally this information given by de doctor (who is usually a resident) and can fall short or insufficient. We hypothesized how a CB could help to give patients adequate information about common surgical ED conditions, and what residents could learn from it. Methods We conducted a prospective pilot study to evaluate how patients qualitatively grade and rate the information given by a CB and by a second-year surgical resident. We wrote a statement where we requested the most relevant information about the patient’s pathology, the therapeutic options and the complications that could occur for 3 common surgical infection conditions (acute appendicitis, acute diverticulitis and acute cholecystitis). We presented this statement to Chat GPT 3.5 (CGPT) and 3 trainees (one for each condition) and asked them to write the information requested in 400 words. The statement should be written in understandable and close language. We recruited 12 patients that read the statements. They received both sets of information (Chat GPT’s and resident’s) before admission, and we asked them to rate in a Likert scale the grade of confidence, clear language, empathy and information content of each of the documents, and if they recognized which of the two was written using AI. Results In the scales, 6 patients (50%) evaluated IA as a better explanation with more confidence, while the other 6 preferred the text written by residents. Resident’s were evaluated as more empathic in 75% of cases. Ten out of the 12 patients (83%) identified the text that had been generated by AI. Analysing the Likert scores in depth, in the confidence category, 2 patients evaluated the CGPT text as ‘very high’, and only 1 the resident’s, however globally confidence was higher for the resident’s texts. Regarding clarity, results were similar between residents and CGPT. Considering empathy, residents were better considered by patients. The quality of information given was considered better for CGPT (8 rated CGPT as high or very high versus 6 for residents). The residents involved in this pilot considered the experience enriching and helpful for their training. Conclusions This descriptive pilot analysis reveals how the AI ​​rates slightly higher in providing information than the residents. This could help identify an area for improvement in their surgical training in this regard, as non-technical skills. Although residents rated higher in empathy, clear language and confidence, we didn`t try to improve this aspects in the IA tool (training the CB), as this could affect the results obtained. It is striking how almost all patients recognized the text generated by AI. CGPT and other AI tools are here to stay and can be a great aid for daily tasks and training.