Can ChatGPT answer health questions in Croatian?

Abstract Background Artificial intelligence has found numerous applications, including in medicine. The use of the ChatGPT large language model, an artificial intelligence tool, is increasingly being considered as a source of information for patients and the general public. Although the model was primarily trained in English, it can also generate responses in other languages. Therefore, the aim of this paper is to evaluate the reliability and readability of ChatGPT when addressing questions related to public health in the Croatian language. Methods The model was asked 23 questions from the field of public health. Two physicians, blinded to each other's responses, assessed the reliability of the answers. Reliability was evaluated by classifying responses as: comprehensive, correct but inadequate, and incorrect. Readability was assessed using a modified version of the Flesch-Kincaid formula. A professor of Croatian assessed the answers in accordance with Croatian language norms. Results The model provided correct or correct-but-inadequate answers to all questions, with no significant difference compared to the evaluators (X² = 1.3939; p = 0.237741). Most responses were categorized as easy or fairly easy to read (74.91%). Compared to English, the responses in Croatian were more readable (X² = 12.615; p = 0.0055). The responses largely adhered to the norms of Croatian spelling, grammar, and vocabulary (74.63%). Conclusions The study suggests that ChatGPT can be a reliable source of information in Croatian for addressing public health-related questions. The responses were mostly accurate, readable, and sufficiently aligned with the norms of the Croatian standard language. With appropriate upgrades related to the Croatian language, ChatGPT could become a useful tool for improving health literacy. Key messages • ChatGPT can be a reliable source of information in Croatian. • With appropriate upgrades related to the Croatian language, ChatGPT could be a useful tool for improving health literacy.