Evaluating Accuracy of AI-Generated Travel Vaccine Recommendations: GPTs in Public Health

Abstract Introduction Artificial intelligence (AI), particularly generative pre-trained transformers (GPTs), holds promise for enhancing public health strategies like travel vaccine recommendations. This pilot study evaluated GPTs’ accuracy in delivering tailored vaccine advisories compared to standards set by public health professionals. Integrating AI’s data analysis capabilities into healthcare decision-making could improve recommendation precision and personalization. Methods This retrospective study analyzed GPT travel vaccine recommendation accuracy against a gold standard from health professionals for 90 travelers visiting various countries. Sociodemographic data and vaccine recommendation outcomes were examined. Accuracy assessments were stratified by age, sex, country of travel, duration of travel, type of travel, and specific vaccines. Results Preliminary results showed an 77.43% overall accuracy across all vaccine recommendations. Specific vaccine accuracies varied, with 97.78% alignment for hepatitis A, 93.51% for typhoid fever and 89.36% for rabies. However, lower accuracies were observed for cholera (33.33%) and encephalitis (56.25%). Moderate accuracies were noted for yellow fever (75%), 74.07% for malaria prophylaxis, diphtheria and tetanus (63.33%), and dengue (57.58%). Stratified analyses are expected to provide more realistic results and highlight discrepancies in certain groups. Conclusions This pilot demonstrates GPTs can be a valuable tool for supporting international vaccination professionals, but they are still not a substitute for these professionals, achieving 77.43% overall accuracy. Recommendations include: a) Educate health professionals in the use of GPTs, b) Developing guidelines for ethical and accurate AI use in public health, c) Using AI to strengthen surveillance and response systems. Combining AI with traditional strategies aims to optimize international traveler health outcomes. Key messages • Using artificial intelligence can optimize and facilitate the task of clinical consultancy in traveler’s advice. • It is necessary to establish guidelines for AI use to ensure ethical and accurate applications.