Improving the Usability of Randomized, Controlled Trials of Artificial Intelligence-based Chatbots in Healthcare: Results of a Systematic Literature Review

Artificial Intelligence (AI)-powered chatbots and virtual assistants (VAs) have gained importance in healthcare to support communication and decisionmaking. The value of Randomized Controlled Trials (RCTs) providing evidence on their effectiveness and safety depends on the quality of their design, conducting and reporting. The CONSORT-AI guideline has been developed to guide researchers in designing, conducting and reporting RCTs for AI interventions. The aim of this research is to identify and analyze RCTs on AI-based chatbots in healthcare, with a special focus on their compliance with the AI-extension part of the CONSORT-AI.A Systematic Literature Review (SLR) was conducted and identified 28 relevant RCTs.Most AI-chatbot application studies were performed in mental health, public health and cancer care.An increase in the number of RCTs, by time, was observed, however, none of the RCTs fully adhered to the CONSORT-AI guideline.Weak reporting quality on input data, inclusion criteria, AI versioning, performance errors and human-AI interactions were common.Improving the Usability of Randomized, Controlled Trials of Artificial Intelligence-based Chatbots in Healthcare: Results of a Systematic Literature Review -146 -Our study highlights that, in parallel with the development of AI-chatbots, from simple statistical models, to advanced neural network architectures like transformers, more emphasis is needed on standardizing their clinical research and reporting to ensure more robust, transparent and ethical evidence for their application in healthcare.