Use of artificial intelligence to transcribe and summarise general practice consultations

Background: This study’s aim was to evaluate the performance of artificial intelligence (AI)-based programmes in transcribing and generating clinically accurate and complete summaries of general practice consultations in real-time. AI has seen many recent advancements, making it more accessible and implementable within the healthcare profession. New large language models (LLMs) such as ChatGPT and BioBART have presented the possibility of developing tools to handle common administrative tasks in general practice, with some options already appearing on the market without any peer-reviewed evaluations. A general practitioner (GP) in the UK, on average, spends 14% of their day on clinical administrative tasks of making entries in electronic health records (EHRs), and greater administrative burden is known to be associated with increased physician burnout. There is also a possibility of recall bias-related inaccuracies in addition to the administrative burden of making entries into EHRs as often these tasks are performed at the end of the clinic rather than at the end of the consultation. We wanted to explore the possibility of using LLMs for summarising GP consultations in real-time, by doing a proof-of-concept study. Methods: This study quantitatively evaluated four selected LLM (some of them were fine-tuned based on a bank of 17,000 patient-doctor dialogues). We used publicly available N=47 simulated GP consultations and applied the selected LLM in generating summaries for these consultations. The accuracy of these AI-generated summaries was evaluated independently by two reviewers independently by a modified form of the Q-note score, which has been previously validated to evaluate accuracy of EHR entries. Results: After preliminary evaluation, a model fine-tuned on GPT-4 was used for the full analysis. The model fine-tuned using GPT-4 produced 47 consultation summaries, with 80.9% of the summaries scoring 46 or above out of a total score of 52. The results were further analysed by dividing the data based on 12 different categories based on the symptom profile of presentations, and no significant performance difference was detected in the accuracy of AI-generated summaries (P value =0.48). The most frequent errors noted were related to failure in recording relevant negative findings, recording of over-the-counter medications, and recording of numerical findings from the physical examination. Conclusions: The cloud-based GPT-4 model was successful in consistently producing reasonably complete and accurate summaries of simulated primary care consultations. Further research is required to address some of the limitations in recording certain aspects of GP consultations and may require further training of LLM to address that. Future research also needs to evaluate the performance of these models in real-life situations which will help further iterations and refinements. Our study, however, is an important first step in safe implementation of AI/LLM in summarising doctor-patient consultations.