A Trustworthy Analysis Approach for Chatbots on Health Data: ChatGPT-4 Example

Considering the increasing and widespread use of chatbots, it is of great importance to provide methods and tools to address ethical concerns and to make users aware of various aspects of a chatbot, including non-functional features. The Generative Pre-Trained Transformer (GPT) is a state-of-the-art Natural Language Generation (NLG) model, enhanced by the latest version of GPT-4. This study evaluates the trustworthiness of the responses of three different chatbot models, BlenderBot with Hugging Face, ChatGPT-3.5 and ChatGPT-4, to 100 randomly selected medical questions. The accuracy and semantic similarity of the answers were measured by BLEU, ROUGE-1 and Cosine Similarity metrics, and response times were also recorded as an important performance factor. The results showed that GPT-4 exhibited superior performance, thus being able to produce more accurate and contextually reliable responses. However, the significantly longer response time of GPT-4 emerged as a disadvantage that may affect real-time utilisation. These findings provide an important reference for the effective use of GPT-4 in the context of health chatbots. The study contributes to the literature on improving the effectiveness of chatbots in healthcare by drawing attention to the importance of balancing speed, accuracy and trustworthiness, as well as the fact that the answers obtained are drawn with Application Programming Interface (API).