Evaluating large language models for postoperative pain education: safety, quality, and readability

Large language model (LLM)-based chatbots are increasingly used for patient-facing medical information, but their performance in postoperative pain education remains uncertain. We evaluated safety, accuracy, empathy, information quality, and readability across seven public-facing chatbot interfaces using 51 postoperative pain questions (357 responses). Safety was assessed by blinded clinical review, and between-interface differences were analyzed using paired nonparametric tests with multiplicity adjustment. Eleven responses (3.08%) were classified as Unsafe after final adjudication. Safe-response rates ranged from 94.1% to 100.0%, with no significant between-interface difference in Safety ( P = 0.617). Accuracy, empathy, information quality, and readability differed significantly between interfaces (all p < 0.001). ChatGPT had the highest accuracy score, Claude the highest empathy score, and Copilot the most favorable readability profile. Public-facing LLM chatbots may support general postoperative pain education, but no interface performed best across all domains. They should be used as adjuncts rather than substitutes for clinician-provided discharge instructions or individualized postoperative care.